• Freedom
• Quality
• Security
• Try it first
• Community

And many others. How about customizability? Have you already realized that some feature is missing in product “foo”? Fine, what now? Well try to bring the vendor to extend it. But wait, we are using open source software, right? Let’s patch it then Let’s see how easy it is … but the problem first:

$>git

usage: git [--version] [--exec-path[= ]] ...

The most commonly used git commands are:

add      Add file contents to the index
branch   List, create, or delete branches
clone    Clone a repository into a new directory
[...]

So now let’s compare that output with our another favorite tool that we all so love: Gerrit:

$>ssh gerrit gerrit --help

gerrit COMMAND [ARG ...] [--] [--help (-h)]
[...]
Available commands of gerrit are:

   ban-commit
   create-account
[...]

Well, almoust identically? What about mandatory “–help” option and missing command descriptions? Yes. It is even more annoying once you extend Gerrit and create a plugin for it, like we did. Gerrit plugin typically exposes some functionality through ssh commands. And the consumer would like to know what these commands are doing. And that’s very handy to have one line description for it. Note: plugin uses the same extended documentation functionality like Gerrit code: markdown html page generation, but you still need a browser to read it. So let’s patch it upstream. It was never so easy with Gerrit: clone it, extend it and upload it to Gerrit, then wait for review. Once reviewed adjust it and wait for merging. Once it is merged, recieve congratulations for extending your favorite tool upstream

$>ssh gerrit gerrit
Available commands of gerrit are:


   ban-commit      Ban a commit from a project's repo
   create-account  Create a new batch/role account
[...]

Is that the whole story? Well not really, how about manual page?

$>man gerrit-review
No manual entry for gerrit-review

Hm… That would be hard, in the end we consume only ssh commands from Gerrit. And Gerrit is shipped with pretty cool documentation. OTOH that documentation is html based and we have to leave our terminal and switch to browser… really? So let’s patch that too. And create local gerrit man-pages. After deploying, it just work:

$>man gerrit-review
[...]

Note: once the core Gerrit provides manual pages, the tool chain is established, it is trivial now to provide manual pages for plugins. Just copy the makefile adjust it, generate manual pages and distribute it. So, once you are using open source and missing something … go ahead and customize it

Many thanks to Norbert who fisrt argued, that Gerrit is missing these features and to great Gerrit community for helping me to integrate it!

.IF "$(BUILD_X64)"!=""
# ---------------- X64 stuff special ---------------------
#  use UNICODE only because shell/shlxthandler
#  doesn't link against ascii_expat_xmlparse
#---------------------------------------------------------
SLOFILES_X64=$(SLO_X64)/xmlparse.obj \
             $(SLO_X64)/xmlrole.obj \
             $(SLO_X64)/xmltok.obj
CDEFS_X64+=-DXML_UNICODE -DCOMPILED_FROM_DSP
CFLAGS_X64+=-I..
LIB1TARGET_X64=$(SLB_X64)/$(TARGET)_xmlparse.lib
LIB1OBJFILES_X64=$(SLO_X64)/xmlparse.obj
LIB2TARGET_X64=$(SLB_X64)/$(TARGET)_xmltok.lib
LIB2OBJFILES_X64=$(SLO_X64)/xmlrole.obj \
    $(SLO_X64)/xmltok.obj
.ENDIF # "$(BUILD_X64)"!=""

So what is BUILD_X64 here and when it is used? My understanding was, that LO isn’t shipping as 64 bit binary on Windows. Tor did some great work on it but it is still highly experimented. So what is all that stuff about? It turns out, that LO does shipp nativ 64 bit extension for 64 bit OS for shell integration and special version of expat (64 bit) must be created for that.

Not big deal, right? So what’s the problem with that? The problem was to test it. Gbuildification has already broken almoust everything, so before merging the gbuildified expat to master some real tests should be done, especially if some native bits are involved. Having only access to WinXP (32 bit) and VS 2008 Express was clearly not sufficient. So before installing a whole blown testbed with new VM, 2008 Server 64 bit, SDKs and 64 bit VS compiler i thought i would just ask the folks out there for help.

So i asked Norbert on LO congress to attach his 2008R2@16 tinderbox to my gerrit patch. He did (thank you Norbert!) and it failed. Unfortunatelly i din’t figure out how to access the error log for that. Well, i configured a new VM and installed Windows Server 2008, SDKs, VS 2008 Express compiler to test my patch. But the disappointment was really big: this configuration neither tested 64 bit shell extension nor did it test expat 64 bit code. As i understand now, VS express edition doesn’t ship 64 bit executables (or at least i didn’t figure out how to install that).

Having access to VS 2010 professional only, i found myself starting a new adventure: adjusting LO code to compile on that new compiler. Thank Peter’s work it almoust worked. I filed some changes in this gerrit patch link and updated the documentation correspondingly: here and here. One weird templating problem in writerfilter is still not solved, though: link. Any help here would be highly appreciated!

After all these tricky problems i could check my patch and found the problem. It was failing to link 64 bit native libs because of architecture mismatch: objects were (still) built with _X86_ but the lib was AMD64. The reason for that was: i forgot to extend the compilation to honour BUILD_X64 setting. So i introduced a new function and used it (in C++ case it was already there) and all was fine:

gb_StaticLibrary_add_x64_generated_cobjects

In the end i wanted to test the created msi file but was experiencing that weird msi installation problem: fdo#55290. It was clear that the problem was introduced with gbuildification of cli_ure module.

The comparison of verbose output of old dmake build and the new one reveals the different approaches in creating the native library. While in the old build system the library was created under the end name cli_cppuhelper.dll and signed afterwards, on the new gbuild system it was first created under the different name cli_cppuhelper_native.dll and then renamed during the signing step to the end name cli_cppuhelper.dll. This is due the fact, that in gbuild world we don’t want to mess with copying artifacts around: all libs are living under bin directory. That why that is natural on gbuild to prefer name differentiation over different directories.

But the problem is that the native library may not be renamed. If you look inside it (emacs) you can see that it “remembers” his name. Like always, the standard action sequence: “reproduce, isolate, find workaround, understand and fix” worked here.

After all, i am able to install a fresh built LibO-Dev_3.7.0.0.alpha0_Win_x86_install_en-US.msi, compiled with VS 2010 and contained gbuidified expat with 64 bit shell integration. But Michael Stahl merged my gerrit expat module already (thank you Michael!) and besides i have no idea how to test that LO shell integration? May be some one can point me to a RTFM or send me a screen shot: it would be nice to know why i was messing around with all that stuff. I wanted only gbuildify expat module …

First time i compiled LibreOffice (LO) in march 2012, it failed to compile in the seamonkey (moz) module (i didn’t enable it explicitly). I had many questions about moz integration in LO:

1. what has an office suite to do with a browser?
2. why it is enabled per default, if it’s something optional?
3. why it was failing to compile on my recent Ubuntu box?

Six months later i knew the answers to all these questions

1. it is needed for Thunderbird address book integration and some other extensions.
2. it is (well was) not really optional: mail merge is an important feature, because sooner or later you want to send mails from your office suite to the recipients from your address book. To import the Mozilla Address Book (MAB) data, which is stored in mork file format, LO need some means to read and interpret that format. Because of lack of alternative implementations Mozilla itself was used for that purpose.
3. well here things started to become complicated. It was subtle linking issue: compiler tool chain configuration was changed on Debian/Ubuntu: “as-needed” became the default, breaking many tools and libraries, among other the moz module in LO. But why two distros had a compilation problem in the moz module and nobody noticed? Debian/Ubuntu maintainers disabled this “a strangely hacked up Mozilla binary” inside of LO.

The chronology of events was:

• All was fine: moz module could be compiled on Debian/Ubuntu
• Debian/Ubuntu maintainer disabled mozilla in LO: “It is NO OPTION to use obsolete, patched seamonkey for providing the Mozilla Adress Book stuff.  [...] Thus it’s (let alone for security reasons) disabled in most distros.” link
• Debian/Ubuntu introduced a compiler tool chain configuration change, making “as-needed” the default (see here and here)
• folks tried to compile LO and it was failing on Debian/Ubuntu

At that stage I didn’t know the whole story but I could convince the LO dev guys to invert the logic and to make –disable-mozilla the default option. Finally I fixed the compilation problem (providing “-Wl,–no-as-needed” option) and forgot about that stuff.

Work on mork driver

Some time later a core developer sent me a link to this bugzilla issue and asked me if I would like to take a look at it. I was interested. Submitter of that easy hack did very impressive job with his in depths description. I started to analyse how all parts were matching together: mozab driver, profile discovery, MQuery, xpcom backend driver (part of seamonkey) and the new open source mork file parser. It was very quickly obviously that substantial part of old mozab driver implementation must be reused. After some deeper investigation of the old implementation and the new mork parser the whole picture was clear to me. Basically there are two use cases:

• MorkDriver Bootstrap:

• Query Execution:

Some words about mork file format. In a nutshell mork file format is a generic sparce matrix data structure. It matches exactly the requirements: in an average address book only a small part of possible properties is actually filled. I you would open abook.mab file, you would see something like that:

[keys]:
  (83=FirstName)(84=LastName)(85=PhoneticFirstName)
  (86=PhoneticLastName)(87=DisplayName)(88=NickName)
  (89=PrimaryEmail) [...]

[values]:
(81=John)(82=Doe)(83=)
(84='John Doe')(85=jdoe@somewhere.org)
[...]

[rows]:
  [1(^83^81)(^84^82)(^85=)(^86=)
    (^87^84)(^88^81)(^89^85)
[...]

Well, there is no magic once you understand how it works. We have overlaping key and value namespaces that makes it some hard to understand. It was used to save the place (for which cost?). So the common construct in mork file is (^83^81). This is a so called cell and consist from pointer to column and pointer to value, i. e. in the record above the name is Joe, the last name is Doe and the email is “jdoe@somewhere.org”.

I sarted the implementation with a small utility: mork_helper.cxx. With it you can dump the whole content of your abook.mab file (using profile discovery to locate it first). Once it worked i shamelessly stollen substantial part of the old mozab driver, replacing more or less the interface to the backend and the backend itsef.

Conclusion:

It was really exciting time to work on new mork driver for LibreOffice. On linux platform the new mork driver would be shipped from release 3.7 on. However on windows old seamonkey binary continue to be packaged. On windows, seamonkey is also in charge for reading Outlook and Outlook Express file formats. There is no native (not deprecated) API for that. That’s pretty uncool. I’m thinking of extracting that part of seamonkey and use it directly (like open source mork parser) without shipping the whole seamonkey. Once in place we would hopefully be able to strip this “strangely hacked up Mozilla binary” from LO repository. And last but not least many thanks to all who helped out during the development and integration, especially Stephan Bergmann, Michael Stahl, Fridrich Strba, Michael Meeks, David Tardon and Caolan McNamara.

scratched

a/ Allow dev to pre-build commit on a variety of platform _before_
pushing them to master. iow check and push rather than the current
push and fix. [...]

To see how gerrit fits in current LibreOffice CI picture or what should be done for it let us first take a look at the situation prior to gerrit migration. At the moment there are 23 Tinderboxes (TB), that independently running on donated machines and asynchronously verify different git branches. They upload the build results to the master TB per mail. Master TB extracts the log from the mail and puts it to the web site for that specific branch, i.e. for master you can find it here.

So basically we have to set up some process let us name it task queue that has to solve the follow challenges to integrate TB and gerrit change verification:

• one gerrit change must ber verifies on variety of platform simultaneously by different TB.
• ACL: only authorized gerrit user might (mostly member of non nteractive group, i. e. BuildBot) be able to do it.
• if more then one TB exists for each platform engaged in gerrit patch verification (and this is often the case)  only one TB may win the gerrit change verification competition. Obviously that require something that behaves like a blocking queue for each platform.
• One combined approval message for all platform has to be reported back to gerrit: +a if and only if all tasks on all platform were successful. -1 otherwise.
• idle-timeout: if task is scheduled to platform specific verification queue and was not picked up. Note: it is not a failure.
• engaged-timeout: task was picked up, but no result was reported during reasonable amount of time. In that case one can think about recovering process: reschedule the same task once time again to platform specific verification queue.
• Receive the log files from TB and present them. It is clear that fire and forget functionality (TB send per mail result to master TB) doesn’t work with gerrit, because the verification message must contain link to the log file and TB master is not involved in that use case.
• Communication between tinbuild2 (bash script) and gerrit itself.

LibreOffice is not the first project that integrates gerrit with CI Infrastructure. We have Jenkins Gerrit Trigger Plugin and Openstack project has zuul. But what makes the situation unique is the fact, that the TB don’t maintain the continuous connection to any master, they appear only for a moment, pick a task, disappear, report results and so on. To address this problem a new approach must be implemented.

Since version 2.4 and upcoming 2.5 gerrit provides mature plugin architecture. gerrit plugin has access to all gerrit services, including

• native support for SSH commands.

• stream event listener provides access to all gerrit events, i.e. patchset created.

• publish event factory used to report review/verify message for gerrit change

• HTTP module integrates with gerrit web application. It is needed to implement access to the log files, received from TB.

To see how it fits all together, look at that picture:

1. Buildbot gerrit plugin listens to gerrit patch set created events.
2. developer uploads a patch set to gerrit.
3. Listener receives the event and creates a GerritJob instance with n TB tasks (one per each platform). Those tasks are maintained in platform specific blocking queue.
4. TB ask per ssh for a task with get-task command (for their platform). The task is removed from the queue and TB is engaged. In that way it is guaranteed that only one TB get the same task to build.
5. Once ready TB reports the result and send the log file and the status per ssh with report command.
6. When the last task status is reported for the gerrit job is considered as ready and combined verify message is reported back to gerrit. The message contains the links to the log files for all checked platform.
7. Behind that link the the own log file servlet is located.

Current status of plugin: basic functionality is provided. Code must be consolidated and more tests and feature must be implemented: Multi project support, Admin GUI, persistence of queue, timeout behaviour. If you want contribute you can find the code … on gerrit: Buildbot-gerrit-plugin. Here are steps how to contribute:

1. register with OpenID to gerrit and upload your public keygerrit:.
2. clone the repo:
   git clone ssh://davido@review.idaia.de:29418/gerrit-buildbot.git .
3. install git-review:
   sudo pip install git-review
   and set it up: git review –setup
4. change something and commit
5. push to gerrit:
   git review
6. wait for verification/approval.

• python-daemon
• python-yaml
• irc

To be configured it needs two files: one for gerrit and IRC configuration and one for mapping of multiple IRC channels to gerrit events. Sample for gerritbot.cfg file:

[ircbot]
 nick=myIRCgerritBot
 pass=
 server=irc.freenode.net
 port=6667
 channel_config=/home/gerrit2/bot/channel.yaml

[gerrit]
 user=gerritbot
 key=/home/gerrit2/.ssh/id_rsa
 host=gerrit.libreoffice.org
 port=29418

gerritbot.channel.yaml:

libreoffice-dev:
 events:
 - patchset-created
 - change-merged
 projects:
 - dev-tools
 branches:
 - master

libreoffice-infra:
 events:
 - patchset-created
 - change-merged
 projects:
 - dev-tools
 branches:
 - master

As you can see, we have configured two different channels and mapped the same gerrit events to both. To start you can use gerritbot.init file or just start it as a daemon from the command line with the gerritbot.cfg as argument. Note: if daemon is not started (lock file is not created) then you may experience some configuration problem. While gerritbot still doesn’t provides a logging machinery (why wouldn’t you provide a patch for that?) it might be manipulated to comment out daemonization and run it as a script. Just comment out four lines and call _main() directly. Now you must see configuration problems.

Another problem may occur with IRC channel. These one can be tracked down with irclib. Just turn on DEBUG=1 in irclib.py on your system and irclib would be very verbose and would tell you the whole truth.

Every time someone upload or commenting something on gerrit IRC bot would say something like that:

logerritbot David Ostrovsky proposed a change to dev-tools: add configuration files for IRC gerrit bot  https://gerrit.libreoffice.org/327