The Qt Messaging Framework, QMF, consists of a C++ library and daemon server process that can be used to build email clients, and more generally software that interacts with email and mail servers. QMF is designed to be extensible to allow other types of internet and telephony messages to be handled. Currently QMF is ported to Linux and Microsoft Windows, other platforms will be ported to in the future. Check out the QMF source on Gitorious
There are currently no plans for QMF to become part of the Qt library, but it may subsequently be offered as a Qt solution. QMF does not provide the ability to create a cross platform application that uses the native messaging system on the underlying platform, such as the Symbian Messaging Framework on S60 or Microsoft MAPI on Windows Mobile. For more information on this topic please see the upcoming Qt Mobility Messaging labs blog entry.
QMF is a fully functional solution and comes with a demonstration mail client, called qtmail.
The QMF library is used by both the QMF message server process and also by QMF client processes. The message server is a long lived process, responsible for performing synchronization with external mail servers, which are normally located on the internet. Clients use the QMF library to interact with the messaging server, internally IPC is used by the library implementation to communicate between the server and clients.
Messaging data is stored locally in a database (currently SQLite). Clients and the message server access the database using the QMF library, specifically through an interface consisting of a singleton class called QMailStore. QMailStore provides shared access to the database by using semaphores, and notifies clients of changes to the database using standard Qt signals.
The message server loads plugins, known as protocol plugins, at startup to communicate with external mail servers. Protocol plugins currently exist for IMAP, POP, and SMTP. New plugins may be implemented to allow communication using other protocols. Like the message server and clients, plugins use QMailStore to access the local messaging database. Additionally plugins use standard Qt network classes such as QSslSocket to communicate with external mail servers over TCP/IP connections.
QMF provides a number of features which are important for mobile computers.
The message server working with well implemented clients should operate within a fixed upper bound of memory. This is made possible by providing interfaces to partially retrieve message data, including both message attachments and message bodies, lists of messages in a folder, and lists of folders in an account. Large messages are efficiently handled with respect to memory by using the mmap function.
Push email functionality is provided as demonstrated by the IMAP Protocol Plugin which allows monitoring multiple mailboxes using IDLE. Additionally a complete implementation of the Lemonade standard is provided including forward without download, and quick resynchronization. Furthermore email metadata can be retrieved independently of messages, this metadata is analyzed to determine which parts of messages to download, and if necessary used to refresh the status of messages as they become visible. All of these techniques help to conserve bandwidth.
Round trips to external mail servers can be very expensive taking several seconds to complete. QMF eliminates unnecessary round trips by fetching data for multiple messages with a single IMAP instruction. QMF is also capable of pipelining commands to reduce round trips but this functionality has yet to be fully utilized.
Additionally QMF provides automatic detection of conversation threads by using messages headers, including the subject header. Conversations spanning multiple folders such as the inbox and sent folder are detected. A full featured searching and sorting interface is present, including multiple levels of sorting, as is a complete Boolean algebra filtering interface, that even allows filtering on custom fields. The disconnected access paradigm is supported by QMF. For a more complete description of QMF functionality see the documentation included in the source repository.
In summary QMF is an advanced mobile messaging framework.
No related posts.
24 comments
So another Akonadi?
QMF is designed for mobile networks and devices. QMF implements the Lemonade Profile, and is optimized for high cost, high latency, low bandwidth environments on devices with relatively small and limited resource capabilities. Akonadi targets desktops.
Thus QMF and Akonadi have different requirements, target audiences, designs and implementations.
If you would like to know more about the challenges of designing a messaging stack that works well on mobile devices there’s a nice article about it here:
http://www.isode.com/whitepapers/iphone-imap.html
Will it support activesync? Are these going to be the basis for maemo, s60, etc?
I hope so! Why not?
However these decisions aren’t up to me, and I don’t have answers to your questions. And even if I did I can’t share future plans of Maemo, S60, Qt Software or Nokia in general.
I apologize for not being able to give a better answer, but I hope this is better than nothing.
Akonadi is not dependent on a regular desktop, technically it can run just as well on limited hardware, especially once the sqlite backend is finished and stable. I see no reason why another such framework had to be created.
Jeez! Yet another “Brisbane reinvents the wheel” framework. This is so full of NIH again…
Andras please see the isode white paper I linked to above for information on why mobile messaging stacks require a different design from desktop messaging stacks. As an example the requirement to support partial retrieval of the list of messages in a folder leads to a different design outcome with regards to synchronization logic.
Blop it’s true Akonadi could and can be re-engineered to support this model. But QMF is based in part on the Qtopia messaging stack, which has been worked on for over 7 years and hence predates Akonadi. So I don’t see how it is valid to criticize QMF for NIH syndrome.
I’m guessing the “QMF API” arrows in the diagram are all in-process api calls? If so I’d agree that its nothing like Akonadi.
Here are my 2 cent with my Akonadi developer hat on…
I don’t think that these projects have something to do with the NIH syndrome but they have been started to fulfill the requirements that Nokia has with Qt for the mobile platforms. Akonadi has been started in the first place to have a solution for KDEPIM, that GNOME or other desktop environments can use it as well doesn’t imply that every PIM application has to use it from now on… We are aware that with MySQL as the only really working cache-storage backend Akonadi is out of scope for most mobile devices (let’s wait some years, maybe the hardware is more powerful then) and the goals of the project (a general, type agnostic PIM access layer) also differs from the goals of QMF that seems to be targeted on email and message handling only.
Nevertheless I see a good potential to add support for QMF to Akonadi (if there should be a use case) or reuse some of the MAPI communication code in the Akonadi Exchange resource.
So blaming the Trolls/Qties/DownUnderies for NIH is pointless on this topic…
Disclaimer: I’m a developer of another Qt-based IMAP product (http://trojita.flaska.net/).
As I’m quite interested in playing with IMAP, I’ve decided to give your client a try. Please accept my apologies if this is not the best channel to report these observations, I’m not aware of any better path (In particular, I was not able to google anything about the “qtmobility” list that the blog post is speaking about — is it a list, or just some internal alias? It isn’t listed at http://lists.trolltech.com/ for sure.).
1) The README file references HTML documentation which gets built only by running the qdoc3 by hand (which in turns requires an evn var to function properly).
.
2) In the example mail application, you offer no way to ask for the password everytime. Given that it is stored in cleartext in the sqlite3 DB (and in its associated journal file during insertions), it’d be great if you offered a way to work without password being stored permanently.
3) `messageserver` segfaulted for me. Where should I send backtraces (if I happen to find out how do I tell qmake to include all symbols)?
4) You don’t seem to decode RFC2047-encoded headers (like non-ASCII sender/recipient names or subjects). Too bad, I wanted to use your code in my application as mine is buggy
5) My laptop disk is ticking like hell when I open my INBOX on a remote Dovecot server. It has just 3206 messages, that should not be that much.
6) This might not be completely fair (see the disclaimer), but I feel that the application is not really “as fast as it could be”. It might be related to the permanent storage you use (haven’t checked, but note that sqlite3′s performance is rather sub-par when you do implicit commits), but my app can open the same mailbox almost immediately (with empty local cache).
Ian the QMF API arrows to the database are in-process. The QMF API calls between the clients and the message server involve IPC, because the clients and server run in different processes.
This IPC consists of light weight messages, called service actions. Like retrieve/send a message with this identifier, or export changes made on the device while offline to the server.
QMF is designed to carefully eliminate unnecessary data transmission via IPC between the clients and server, because IPC data transfer is expensive. Instead the server updates the database directly. And then the clients read from the database.
Database is a bit of a simplification. Message bodies and attachments are stored using the file system, and only message meta-data is stored in the SQL database (subject, date, sender, recipients, conversation threading info, file paths to message body and attachments, …) .
For data on the file system (outside of the SQL database) mmap is used to keep large chunks of data in kernel space, so that they don’t leak into user space via expensive copying.
Also for pedantic readers, QMF uses semaphores to manage concurrent access to the sqlite database, which is technically a form of IPC.
BTW QMF isn’t tightly coupled with sqlite. Qt SQL classes are used to provide a layer of abstraction between QMF and the SQL database.
I hope this helps you to determine how like QMF is to Akonadi. Admittedly I’m not an Akonadi expert.
Hi Jan,
Firstly thanks for your high quality post, it looks like exactly the kind of feedback that we hope for with these labs blogs. Thanks!
Regarding your points:
1) Yes, that is correct. I acknowledge it’d be nice if html doc was included.
2) It’s true the demonstration application doesn’t provide an option to prompt for the password. The demonstration application has many limitations. Now regarding the password, to avoid storing the password it’s necessary to reimplement QMailAuthenticator http://qt.gitorious.org/qt-labs/messagingframework/blobs/master/src/libraries/messageserver/qmailauthenticator.cpp
3) I’m interested in your backtrace and any info on reproducing the bug. You can send them to me for now don.sanders at nokia dot com.
4) RFC2047 should definitely be handled. See src/libraries/qmailmessage.cpp decodeWord. If you have a mail that isn’t being decoded correctly please send it to me (as a tared or zipped attachment).
5) Ideally the number of messages in the folder should not change the performance of the client, especially for the first sync. If all 3206 messages in the inbox are being downloaded by qtmail that’s a bug, in qtmail only the first 20 should be downloaded by default. I’m interested in a bug report on this also.
6) qtmail/qmf uses the disconnected access model. This requires downloading IMAP envelopes for all mails and bodies and attachments for small mails, also conversation threading requires some CPU overhead, and commiting to the database requires a fsync. Where as online clients can be faster as they only need to download headers.
QMF is optimized for slower links. But I think QMF is not terribly slower than other disconnected mail clients.
Having said all that message downloading speed is extremely important. So thanks for mentioning this point, and please feel welcome to email me and let me know about your experience (performance measurements). If you want a fast fix you can try something like a query.exec(QLatin1String(“”PRAGMA synchronous=OFF”")); around line 2127 in src/libraries/qmailstore_p.cpp and commenting out any fsyncs in the code, at your own risk.
Regarding the high disk activity, are you sure you aren’t getting chomped by the firefox famous sqlite fsync error:
http://lwn.net/Articles/283745/
where every call to fsync on ext3 flushes all pending buffers to disk?
Right, I’ve been unfair here, it’s not NIH, it’s about recycling the dead and low quality Qtopia stuff.
Hi
i have two questions :
1-The data base part will be flexible or juste INBOX architecture should be use??? For me, the INBOX architecture is not adapted for the future use because a lot of people have more of one e-mail. And the centralization of all of this will be important. For example, the possibility to use a label mail to regroup mail throw all managed mail.
2- Now, more and more pc should be mobile and it will be connected on different FAI. Unfortunately, smtp is not adapted. For example, if you use mail from FREE fai, you can use smtp server only if you are connected on this FAI. Have you think about possibility to resolve this?
@mongaulois:
Hi, sorry if I haven’t interpreted your questions correctly but:
1. The data model is flexible – you can have multiple different accounts, and you can arrange their messages into a logical grouping arrangement that suits you. The library class QMailMessageSet is provided to allow message grouping elements to be defined (for folders, ‘virtual’ folders or other logical groupings, such as tagged sets). The QMailAccountMessageSet, QMailFolderMessageSet and QMailFilterMessageSet are predefined to make this facility easier to use. The folder model (or logical grouping of messages) used in the qtmail example application is defined in the src/libraries/qmfutil/emailfoldermodel.* files.
2. If FAI means ‘Internet Service Provider’, then no – we have no handling for this issue directly. An extra layer of logic is required to make the connection between a particular SMTP account and the provider to which a device is connected. The may also be some relevance here for the bearer management facility that is part of the QtMobility project.
Using sqlite seems a bit difficult compared to an already IPC db like PostgreSQL I have to say.
But thats just implementation detail, sqlite seems to be considered the gold standard of the mobile world in general (eg Tracker uses it as well).
Thanks for your clear responses! I’m no akonadi expert either, but basically Akonadi communicates to clients in a binary IPC protocol (based on IMAP). The storage backend is entirely internal to the akonadi process. So QMF and Akonadi are trying to solve the same problem: one process handling email communication, but allowing multiple clients. Just does it very differently.
@MattV
1- a good example is gmail where all mail are centralized and you can apply different labels on the same mail. I don’t think you can do this with a folder model. It just another philosophy.
2- oups FAI is in french.. ok. A lot of (all?) ‘Internet Service Provider’ give a free smtp which can be used without login/password if you are connected behind it. If you have more of one mai account, you could use smtp of another mail (which work) to send all mail. So, possibility to change dynamically smtp could be good.
thanks.
I try to build it with qt 4.6 tp1 and visual 2008 tp1 :
* build failed. Don’t find moc file…
* How to product doc?
I have some Qt code to talk to exchange servers using webdav. Is there any documentation to write a server plugin?
@Mongaulois
1. You can build something along the same lines as gmail using QMF. You would use the custom field facility to associate labels with messages, by some internal logic. Then, instead of building a hierarchy of folders, you would create a list (or hierarchy) of message sets, each filtering for messages having a particular label. You can do this with the QMailFilterMessageSet class, using the QMailMessageKey::customField(…) selector.
2. Yes, dynamically selecting SMTP providers would be useful in this scenario. To do this with QMF, you would create an SMTP account for each FAI that you wanted to use. Then you would need to dynamically alter the QMailAccount::Enabled status of those SMTP accounts as you up/down the relevant interface.
Regarding the build issue, please provide some more detail on what you were trying to build.
Building may be easier once you have read the docs, which can be generated by the following command IIRC:
$ THISYEAR=2009 qdoc3 /doc/src/qmf.qdocconf
In that last line of the previous post, the path argument should have a prefix corresponding to wherever the QMF files are, of course.
@rupeshk_19
No, there is no ‘how to do [some task]‘ documentation at this time.
To create a plugin for the QMF server, you need to create a shared library containing a class derived from QMailMessageService, and a corresponding class derived from QMailMessageServicePlugin that contains the boilerplate code that identifies your service. Both of these base classes are defined in src/libraries/messageserver/qmailmessageservice.h (note that server-side code such as plugins needs to link to both the ‘qtopiamail’ client-side library and the ‘messageserver’ server-side library.)
If your service will provide access to external messages, you need to create a subclass of the QMailMessageSource class, and expose that via your service class’ ‘source’ function. If your service will provide a message transmission facility, you need to create a subclass of the QMailMessageSink class and expose that via the service class’ ‘sink’ function. Your service can provide either or both of these.
The existing POP3 plugin in src/plugins/messageservices/pop/ is an example of a fairly simple source service, and the SMTP plugin in src/plugins/messageservices/smtp/ is an example of a sink service.
To make your service configurable by the example settings editor (examples/settings/messagingaccounts), you should create a subclass of QMailMessageServiceEditor which manipulates the configuration data for your plugin, and a subclass of QMailMessageServiceConfigurator which allows your editor class to be instantiated. The existing plugins all contain an example of how to provide these settings.
Hope that helps!
@rupeshk_19
I posted a reply to your question which appears to have triggered a spam filter, or something like that…
If it doesn’t turn up soon, I will try to post again
Hi,
Great to have this product. A remark on docs mentioned by Jan. In order to build html docs run next command:
env THISYEAR=”$(date +%Y)” qdoc3 doc/src/qmf.qdocconf
After that open generated doc/src/html/index.html file in the browser and click “Building and Running”. Now you get the instructions on how to build and run applications.
Comments on this entry are closed.