It is raining massively, outside, again. It does that every day here, in Manaus, what with it being the rainy season and this being the Amazon jungle. The negativity ends there, though, since it takes about 15 minutes, is very refreshing, and everything else here is Awesome (TM). I have really enjoyed the past few days, Bossa Conference has been a great experience. The presentations were generally of high quality, I had many very good conversations over many excellent meals, and by a luxurious pool, met several impressively talented individuals and the equally impressive INdT teams. There is a lot of very nice work being done here in Brazil in Free Software in general and around Qt and KDE in particular. I'm proud to have been invited to come.

Looks like Microsoft has released the PST format specification.

I don't normally like to link to MSDN, but I'll do it this once:
http://msdn.microsoft.com/en-us/library/ff385210.aspx

As usual with these documents, I recommend reading the PDF version rather than the HTML. Also, Firefox seems to handle MSDN a bit better than my (KDE 4.3.5) Konqueror.

If you were a mad-keen PIM hacker, and looking for a GSoC project, might be worth a look.

[Thanks to Tom Devey for the heads-up on this]

It's that the year again when KDE PIM developers attend the annual meeting in Osnabrück, traditionally hosted by Intevation, one of the companies which continously excels in acquiring funding for KDE related development.

Tom already blogged about it and there will be a Dot article as usual.

Unlike other developer meetings such as the Akonadi sprints, the Osnabrück meeting doesn't involve a lot of hacking, for some of use no hacking at all.
It has more a strategic character, i.e. where we evaluate where we stand and what we want to achieve over the next year(s).

There are a lot of awesome developments in KDE PIM space about to unfold in the coming year, proving the value of Free Software for tasks which involve personal data, secure communication, privacy, etc. and that this does not necessarily come at the expense of stability, professionality, or even "bling".

Speaking of bling: PIM is usually not something anyone would associate with bling, especially since the underlying technologies and standards are sometimes hard to understand, old, broken but necessary for compatibility, etc.

However, through the introduction of Akonadi, we now remove these obstacles from the application level, opening it for developers who are more interested in doing new and cool alternative user interfaces.

We will keep working on the traditional applications because there are a lot of user who prefer those, but this shouldn't stop e.g. developers with a Plasma background to do a fully Plasma based PIM experience.
If this sounds like something you want to try, we certainly can and will help you with low level parts such as data engines or services.

One of the great things about KDE 4 is how powerful the APIs for the central components are. In particular, Akonadi and Nepomuk have become very easy to use in custom software and third party applications. I recently discovered another very powerful set of libraries: the plugin API for Koffice. Using those libraries, I recently wrote a little "docker" that lets you attach the documents you are currently working on in koffice to a new calendar event which can be used by any Akonadi-enabled application.

Some of Akonadi's resource agents (usually just called resources) work on local files, some on files containing more than one data object, some on directories containing one data object per file.

For example the "VCard Resource" has one vcf file to work with which in turn contains any number of vcards, i.e. contacts.

Those single file storage containers have a couple of things in common so of course we want to share as much of code between their respective resources as possible.

Unfortunately the data inside the different files is formatted quite differently, so the parsing and creation of C++ objects is rather type specific.
Another difference which can make common code difficult is the expected size of the files and each of its entries, i.e. a VCard file will most likely just contain a couple of dozen contacts, maybe in the lower hundret range while an MBox file can easily reach several thousands of entries and each of its entries (emails) is usually larger than one of the contacts.

So code that works well for contacts or calendars does not necessarily work well for messages and vice versa.

While thinking about possible ways to improve our situation I had the idea of using the same level of abstraction we are already using in Akonadi, i.e. generic "Items" which hold the type specific data as their "Payload".

In other words, if we had an "Akonadi Item File" we would get Akonadi items out if it and not have to care about whether those items transport contacts or emails.

This would still leave us with the problem of different file and item sizes. Again I decided to use concepts already proven useful in Akonadi: payload parts and gettings things on demand.

Payload parts refer to a concept where allow for a payload (remember that could be a contact or an email, etc) be split into parts that make sense for the respective data type, e.g. splitting an email into "Headers", "Body" and so on.

Getting things on demand refers to get whatever parts of an item you are interested in at any time, e.g. only getting headers when listing a mail folder and getting the rest when displaying a selected one.

In Akonadi we do this through jobs and telling those jobs what we expect them to return to us.

The respective Akonadi code would look similar to this:

const Collection collection = someModel->selectedCollection();

ItemFetchJob *job = new ItemFetchJob();
job->fetchScope().fetchPayloadPart( MessagePart::Header );

connect( job, SIGNAL( result( KJob* ) ), this, SLOT( collectionListed( KJob* ) );

with collectionListed() doing something like that

ItemFetchJob *job = dynamic_cast( job );

const Item::List items = job->items();
// fetch the first item as a whole

job = new ItemFetchJob( items[ 0 ] );
job->fetchScope().fetchFullPayload();

connect( job, SIGNAL( result( KJob* ) ), this, SLOT( itemFetched( KJob* ) );

In order to do something similar with files I came up with a concept I called the Akonadi Filestore.
The main interface looks like this (omtting some of the methods not important for our example above)

class StoreInterface
{
  public:
    virtual Collection topLevelCollection() const = 0;

    virtual ItemFetchJob *fetchItems( const Collection &collection, const ItemFetchScope *fetchScope = 0 ) const = 0;

    virtual ItemFetchJob *fetchItem( const Item &item, const ItemFetchScope *fetchScope = 0 ) const = 0;
};

Assuming we have an implementation that operates on a file which contains messages, e.g. an MBox store, we can implement the example above quite similar to the respective Akonadi code.
(note: all jobs are named like the ones from Akonadi but live in their own namespace. Collections, items, fetchscope are directly the classes from Akonadi)

// lets assume mStore is of type StoreInterface* and has been
// initialized properly with an MBox store implementation

// lets list the mails in the stores top level collection
ItemFetchJob *job = mStore->fetchItems( mStore->topLevelCollection() );
job->fetchScope()->fetchPayloadPart( Message::Header );

connect( job, SIGNAL( result( KJob* ) ), this, SLOT( collectionListed( KJob* ) );

with collectionListed() doing again something like that

ItemFetchJob *job = dynamic_cast( job );

const Item::List items = job->items();
// fetch the first item as a whole

job = mStore->fetchItem( items[ 0 ] );
job->fetchScope().fetchFullPayload();

connect( job, SIGNAL( result( KJob* ) ), this, SLOT( itemFetched( KJob* ) );

As you can see we get on demand, payload part fetching with the only type specific thing being the name of the header payload part.
In Akonadi resource this difference would removed because Akonadi forwards the payload part from the client requesting it, so the resource does not have to know any of those identifiers itself.

This is already quite nice but it gets better
- We no longer require the items to be of the same type, say a ZIP file containing contacts, calendars and emails (or like an Outlook PST file)

- We can include metadata that is not part of the common payload formats, e.g. flags of email messages if the store's format supports that (or if it is actually working with more than one file, e.g. KMail is saving this things into index files it keeps alongside the actual mail files).

Neither design nor implementation are fully production ready, I consider it a testbed for the concepts mentioned above.
You can find it in PIM Playground (to get correct paths either checkout playground/pim/akonadi or create a directory "akonadi" into which you checkout "filestore")

In an attempt to follow up on my blog about Akonadi porting xplained I am going to write about Akonadi migration.

It is basically the data storage related cousin of porting:
Porting is, as we learned, about adapting applications to a new way of handling data.
Migration is about adapting data to new ways of being accessed.

The last couple of months I unfortunately had too little time for development on either KDE or Akonadi so I spent the available time on thinking about mail migration.

You see, mail is a special case when we talk about migration because it has a couple of key differences when compared to other data types:

- amount
- location
- state/properties

The difference in amount of data is the most obvious one. Most users will have more messages in mail fodlers than contacts in their address book, messages will on average be a lot bigger than contacts, e.g. due to containing attachments.
It is actually very likely that the amount of mail data is several orders of magnitudes greater than that of contact data.
For example I have probably around 100 contacts in my address book and probably around 100 000 messages (if not more), consuming several Gigabytes of disk space.

The different in location of data is referring to the likeliness of messages being stored on a server vs. other data types like contacts.
Recent developments like Google Contacts have shifted that somewhat but it is still more likely to encounter a setup with contacts being locally and mail being remotely stored.
Due to this almost inevitable remote storage any migration process will have to deal with local caching of some sort, e.g. in the case of KMail the maildir used to cache mails of KMail's "Disconnected IMAP" account type.

The difference in data state or properties is a lot less obvious than the other two.
Even in the most basic usage scenario we have the message state changing from unread to read but it is highly likely that messages have additional properties attached such as "important" or "you have replied to this one".
This alone wouldn't be a problem if it would be part of the message or at least part of the file the message is stored in. However not all on-disk formats support that so applications like KMail had to find alternative ways of storing this additional information, e.g. "index" files.

Lets have a look at a couple of scenarios to see how these influence the miration process.

For comparison we'll take migration of a local address book:
- get location of address book file (e.g. KDE's std.vcf)
- create Akonadi storage handler for a VCard file
- point it to the location of the file
- done

A similar approach will work for a local maildir directory:
- get location of the maildir root directory (e.g. $HOME/Mail, $KDEHOME/share/apps/kmail/mail)
- create Akonadi storage handler for MailDir
- point it to the location of the file
- done

The messages stay right where they are, so amount of data is irrelevant. MailDir can encode most of the state data into the file name, so not an immediate problem either. Storage is local, no server involved, no caching to deal with.

Now have a look at another form of local mail storage: mbox
- get location of the mbox file (again rather trivial)
- create Akonadi storage handler for MBox
- point it to the location fo the file
- done

Done? Not quite. Doing it that way we'll lose state data in a way that's probably not acceptable to our users.

So what could we do?
One option would be to make the Akonadi storage handler for MBox understand, e.g. KMail's index files, but that is quite ugly, involved maintaining old code (at least the reading part) and is KMail specific or requires the Akonadi MBox resource to understand all kinds of such additional files.

I'll get back to this later but lets have a look at another example first: mbox file within a maildir tree
- get location of the mbox file (again rather trivial)
- create Akonadi storage handler for MBox
- point it to the location fo the file
- done

Since this shares the same problem as stand-alone mbox, I'll skip the related problems.
However, we have some additional issues here, one being that the Akonadi MBox resource will create a top level folder in Akonadi, thus "moving" the mail box folder out of the tree while keeping the file in it.
One possible solution would to have a resource which can handle mboxes inside maildir trees, but since mbox folders and maildir folders behave differently (e.g. mbox folders need to be "compacted" to really delete mails) we don't consider this a proper solution unless we run out of alternatives.

To not forget about the server location problem, lets finally also look at disconnected IMAP:
- get server connection values and login credentials
- create Akonadi storage handler for IMAP
- tell it about server and user
- done

As an attention paying reader you already know that we are not quite done yet
So what is it this time?
Can't be state of message, everything is on the server. Can't be the resulting folder locations, IMAP servers have always been treated separately.

Obviously it has something to do with the "disconnected" part, so lets have a closer look at that.
It means that KMail has a maildir tree somewhere that is more or less a copy of what's on the IMAP server.
More or less because it is like synchronizing between a remote and a local directory, i.e. changes on either side are not immediately visible on the other side, they are applied at resychronization times.

This leads to two complications for the migration process:
- the users will be very angry if we have them download all those message again
- some messages might have been added or deleted locally and the respective changes have not been synchronized yet

Again a possible solution would be to make the Akonadi IMAP resource understand this local cache and transaction logs, but again this is not a very clean solution.

I am sorry that this is already quite a long blog but maybe you are still interested in some of my thoughts on how to make this work nevertheless.

If we treat the process more like a form of importing instead of simply reattaching different storage handlers, we gain the possibility to change format and locations in a way that allows us to inform the users about these changes and most likely also allow for an advanced mode for people with really specific needs.
So instead of silently doing things in the background, KMail2 could bring up a GUI saying that it has detected a KMail1 setup and lets you choose between ignoring that, importing that the way it sees fit or letting you switch to an import GUI for customization.

Our scenarios above can then be handled like this:

MailDir: no difference there but potentially allowing a customized import routine to move the messages to a new base directory

MBox: top level mbox files can be handled by the Akonadi MBox resource, the importer can be KMail specific and understand the index files, attaching the additional information to the resulting Akonadi message items.
MBox files in side the maildir tree can be read by the importer and added as a new folder to the top level folder managed by the Akonadi MailDir resource, potentially allowing a customized import routine to move the mbox file and treat it like a top level folder instead.

Disconnected IMAP: similar to the "in-tree mbox" case, the importer can be made to understand KMail's form of caching and transaction state handling. However, differently to the "mbox -> maildir folder" conversion, we do not want the Akonadi IMAP resource to add the messages on the server, most of them will already be there.

Again my main idea is to let the importer understand what it is actually importing, in this case how the message is addressed on the server, and attach this information to the message when adding it to a folder managed by the Akonadi IMAP resource.
The IMAP resource will therefore only have to be extended to understand this additionally attached information, not how that used to be stored on disk.

I hope to have some time during the Chrismas holidays to experiement with some of the ideas.

For quite some time almost every blog by a KDE PIM developer is about Akonadi in one for or the other, often about "Akonadi porting" or "porting to Akonadi".

Akonadi itself can already be difficult to explain, combined with "porting" it probably has only meaning left if you are a developer.

The thing anyone else will be able to extract are delays.
Delays in the sense that we, the developers working on KDE PIM, are enthusiastic about our plans on what we want to achieve for a certain KDE release, only to be disappointed that we didn't get as far as we originally hoped for.

For example an initial guess was to have KAddressBook and KOrganizer ported in time for the 4.3 release, but lack of resources meant we didn't even get to start working on KOrganizer in any significant form and the new KAddressBook would lack some features.
So KAddressBook is held back and finished for 4.4, KOrganizer is still being worked on.

So what is all this porting business actually about and why does it take so long?

Imagine a situation where you want to cook something (well, you can imagine anything els of course, but it will greatly improve the understanding of the following example if you restrict your fantasy to cooking for now).

After you've decided what you want to cook, you'll have to get your ingredients, which usually means going out for shopping. When you get home, probably after quite some time and travelled distance, you'll have to prepare these ingredients (wash, peel, chop, etc.) and then start the actual cooking.

That works pretty well, millions of people do between that once in a while and every day.
So you meet other cooks and discover that while you have individual styles for preparing ingredients and the cooking process, you all despise the shopping chore, especially when shops are crowed, roads are jammed, weather is nasty, and so on.

On one of those lazy sunday afternoons you read about outsourcing and immediately convince your fellow cooks to outsource shopping to a shopping specialist.
You snicker at the thought of this poor fellow having to brave the nasty weather, travel the jammed roads, wait in line at stores, while you comfortably wait for the devlivery.

However, you discover that you have to adjust your approach to cooking to accomodate for this for of ingredient acquisition.
With your initial approach you knew when you would be doing what for how long (not counting unavoidable delays during shopping).
With your new approach you don't anything between placing your order and the arrival of the goods. Sure, you could be waiting at the door step, but that gets boring after a while. You can spend some time preparing everything you already have at home, but usually that won't keep you busy long enough. So you do something else instead, maybe reading Planet KDE and learning that those lazy developers have balantly copied your outsourcing idea

Switching to a different reality there is this cook called KMail (in this reality cooks are, for an unknown reason, actually called "mail user agents").
An absolut expert in preparing an ingredient called "message" (weird reality, I know), peeling of envelopes, chopping into pieces (jokingly called "multi parts"), etc. You know, the usual cooking stuff, just done expertly.

But again the shopping for "messages" at shops called "servers", with owners of sometimes questionable character, low quality offerings and so on, is making KMail's life unpleasent.
So of course outsourcing that to a shopping specialist (in this reality, weird as it is, called "Akonadi") is the way to go.

Unfortunately our cook KMail discovers, while adjusting to the different cooking approach, that all these years of shopping had resulted in acquiring certain habits (in another reality, the author of this blog has the habit of not using shopping lists because eventually he'll recognize things to buy at the shop anyway).

Habits that need to be unlearned or replaced by habits which fit the new situation better. Habits our cook might not have been aware of or vowed never to think about again.
Fortunately there is a group of motivational trainers (having the surprisingly sensible name "KDE PIM Developers") who will help to overcome those.

It takes time, but it is totally worth it.

If you have no idea what this means, don't worry, neither do I.

What I do know, however, is that a lot of people around KMail and are extremely happy about this

Basically the folks working hard on porting KDE PIM apps to Akonadi have reached one of their bonus mission goals: they've got rid of a very old, very obscure, tedious to maintain, mindboggling to work with (you get the picture, right?) legacy part of the mail handling framework.

Well, I guess bonus goal might give the wrong impression here. It is not less important than any of the primary mission objectives (to stay in game jargon). It is a bonus because it makes other things easier, boost your enthusiasm, lets you look forward for the challenges to come.
Think in terms of finding the double barrel shotgun in Doom, getting the Holy Grenade in Worms, looting an epic item in WoW, getting super cow powers in Aptitude.

So, congratulations to the folks at KDAB. You rock!

On a related (to my other blog post today) note, while I'm giving credit where credit is due: my personal KDE hero at the moment is Anne Wilson, who has been helping KDEPIM users for years on our lists and at meetings and has been a voice of reason, courtesy, constructive feedback and positiveness that makes a huge difference in the atmosphere of our community. I much admire her work with the documentation team (userbase, anyone?) and the community working group and ever since I first met her in person (in Glasgow, I think) I have been impressed by the fearless and all embracing manner in which she has found her way amongst us weirdos and become a gentle, well respected leader and wrangler of geeks. I don't know when exactly it is, but happy 70th birthday, Anne, all the best from us PIMsters, we thank you and look forward to many more Akademy meetings with you.

With all the excitement and energy surrounding Akonadi and the ongoing porting of our main applications to it at the moment (over 100 commits to KDEPIM yesterday alone!), it's easy to get the impression that we've collectively abandoned our stable versions and the many users relying on them today. Not so. While Volker Krause and his team at KDAB (currently Kevin Ottens, Frank Osterfeld, Sebastian Sauer, Leo Franchi, Stephen Kelly and Laurent Montel, with various others pitching in occasionally, like Marc, Guillermo and Romain) are ripping through KDEPIM trunk, Allen Winter and Thomas McGuire (again aided by Marc and others) are faithfully watching over the stable branches. They are making sure that all relevant bugfixes found by the Akonadi port make it back into the 3.x and 4.x stable branches and are doing many bugfixes and features in those branches themselves, every week, which are then merged into trunk. This results in a steady stream of improvements into both the 3.x and 4.x series, all of which make it to our users (i.e. you out there, probably) via the Linux distributions and via the KDE Windows and Mac packages regularly. This is mostly unglamorous and sometimes boring work which they carry out with great professionalism and personal commitment, both during their KDAB work time and well beyond, in their personal time. They hardly ever get any recognition for what they do, so this is an attempt to remedy that a bit. Rock on, boys!