The end of GSoC 2013

This is the final post for my Google Summer of Code 2013 project – implementing advanced statistics importers for Amarok. Please read the first post if you would like to know more about the project.

So here came the end of Google Summer of Code 2013. It’s the best moment to talk about what has been planned, what was done, and – traditionally – what the future holds. Oh, and let’s get some stats and pictures (because that’s what we’re here for)!

Plans

I planned to:

  • rewrite following statistics importers based on the existing StatSyncing framework:
    • Amarok 1.4 (FastForward)
    • Apple iTunes
  • create new statistics importers for the following media players:
    • Amarok 2.x
    • Rhythmbox
  • soft goal two-way synchronization for created importers (I guess I shouldn’t call them “importers” anymore, but something like “synchronization targets” just sounds too unwieldy)
  • softer goal write tests for created importers
  • side goal make new importers easy to write.

Reality

We all know that thing about plans and reality in software development, right?

©2005 Paragon Innovations, Inc. I promised you pictures.

©2005 Paragon Innovations, Inc. I promised you pictures.

So let’s recap what I’ve done:

  • I created a framework for importers that takes care of managing them, loading and saving their configuration
    • the consequence is that a user can create, remove, reconfigure new importers on a whim
    • programmer can also make as many configuration options as he wants and store as much state in the importer as needed
  • I created following statistics importer types, loadable as plugins:
    • Amarok 1.4
    • Amarok 2.4
    • Apple iTunes
    • Banshee
    • Clementine
    • Rhythmbox
  • …and I implemented two-way synchronization for each of them
  • I wrote tests for every importer and all parts of the framework (except for one helper, as it’s destined to be removed)
  • I made new importers very easy to create – and they automatically gain all goodies from the framework.

Please take a look at earlier posts if you’re interested in pictures – I didn’t want to overload this one with duplicates.

Statistics

Let’s get the total number of changes reported by git:

That’s a nice number of changes. Here’s what it actually sums up to:

The difference between these numbers tells us a story: at least about ⅖ of the code I wrote was removed afterwards through heavy refactoring. (Alright, maybe this wasn’t much of a story.) In other words I not only overshot my initial plans, I was also making sure I’m doing it with style. ;) Of course the numbers take into account only the changes that were finally committed – there were a lot more in between.

Let’s make some code size comparisons between old and new importers.

I really wanted some graphs in this post.

I really wanted some graphs in this post.

As you can see, the numbers are similar. Rhythmbox and iTunes importers are made bigger by XML-processing code (oh how I hate it), and Amarok 2.x and FastForward importers by custom, rich configuration widgets. The simplest importers, Clementine and Banshee, are small and pretty.

Oh, but that’s not the whole story here, is it? All of the new importers also contain write capabilities – they can sync the statistics back to the foreign media player. Without it a new importer can easily fit inside 100 lines, as demonstrated in one of my previous posts. Mission accomplished.

As an aside, I find it interesting to note that the number of lines does translate very well into the number of characters. The average number of characters per line for measured code is 38.17, with standard deviation of 2.60 characters between files.

Oh, and I have proof that I was actually doing something during GSoC. Do take a look at this video that I made, if only for the amazing soundtrack (720p recommended). For details, please see the video’s description.

The future

So, it’s the end of Google Summer of Code, but it’s not the end of the project nor my contribution to Amarok. Arguably the most important event in the project’s lifetime – code review – still lies ahead. Not only that, but I already have some further refactorings in mind.

Other than the project, there’s always a lot to do for Amarok, and the great community around it makes it hard to leave – so I’m not. There’s just too much fun to be had. ;)

Well, that’s it for the post. I’m going to take a few days off and then an academic term starts, and I’m back to my daytime job – I’ll need some time to adjust my schedule. Thanks for sticking around. It’s been – and continues to be – a pleasure!

GSoC Status Update – Week 11 & 12

This is a status update for my Google Summer of Code 2013 project – implementing advanced statistics importers for Amarok. Please read the first post if you would like to know more about the project.

Last week I asked my mentor if I could skip that week’s report and make the next one (i.e. this one) a double one instead. The reason for this was that I was working almost exclusively on tests, and tests more often than not make for a dull post.

Well, after two weeks I have even more tests. I’d go as far as to say that things are satisfyingly tested.

I made a base test suite that relies on convention; importers are expected to have tracks with certain metadata in their databases, and then tests make sure that this data is imported correctly. Tracks with the right metadata are pre-created and stored in the source tree so creating a test database is a matter of adding them to media player’s collection.

There isn’t much more to tests that’s interesting, so let’s skip to the next topic. GSoC 2013 is in the homestretch. September 16 is “soft pencils-down” and September 23 a “hard pencils-down” date. We’re expected to have all the code done on September 16, and to spend the following week on documentation. Having that in mind, this is my much-more-detailed-than-usual plan for this week:

  1. Monday Amarok 2.x importer two-way synchronization (done)
  2. Tuesday FastForward importer two-way synchronization
  3. Wednesday Banshee & Clementine importers two-way synchronization
  4. Thursday Rhythmbox importer two-way synchronization
  5. Friday iTunes importer two-way synchronization*
  6. weekend more tests! Supplement existing test suites

Since I’ve been documenting as I went, the week between September 16 and 23 will be devoted to small bugfixes, design tweaks, typo fixes. Also that’s the week where I remove old media-player importing capabilities and take a minute or two to celebrate.

As always, you can check out my progress on my public Amarok clone. The branch is named gsoc-importers.

Thanks for reading!

* iTunes database will then have to be imported back into iTunes. But hey, it’s synchronization. Kind of.

GSoC Status Update – Week 10

This is a status update for my Google Summer of Code 2013 project – implementing advanced statistics importers for Amarok. Please read the first post if you would like to know more about the project.

Fixing

Last week I fixed Amarok 2.x embedded database importer. There were quite a few problems with handling an external database process:

  • if the QProcess object received commands (start(), kill(), terminate()) from a different thread than that which has created it, it resulted in wrong behavior, often in the form of a crash (can’t get much wronger than that)
  • the QProcess object does not encapsulate the process; it’s only an interface. When a QProcess object was destroyed while the process was still running, it issued a warning and killed the process with SIGKILL. Normally you’d want to stop an ongoing process with a SIGTERM, so manual lifetime management is needed
  • if the server has stopped (which it does, after a period of inactivity), calling QSqlDatabase::removeDatabase() resulted in a SIGPIPE signal from inside the MySQL driver
  • related to that, an old QSqlDatabase connection silently failed to work if the server has been restarted. There would be no warnings on QSqlDatabase::open().

There was also a question of waiting for mysqld process to be ready to serve. After some research I decided to adopt an approach that MySQL startup scripts (including the “official” script, mysql.server) use, which is to wait for the server’s PID file to be modified. Overall, I’m quite satisfied with the results; it’s reliable and fast enough.

Creating

By the way, there’s a new statistics synchronization target: Clementine.

clementine importer

I know how much you like pictures

This brings the total number of importers to six, and marks the end of implementing new importers. From now on, I intend to focus on existing code including, but not limited to, read-write capabilities and – of course – testing.

Simplifying

You may have noticed one additional target on the screenshot above: “Example.” Another thing I focused on this week was code deduplication and simplifying creation of new importers. To show off, I prepared a basic “Example” importer. Below is the full C++ code. Bear in mind that aside from the code, importers need a plugin’s *.desktop file and a CMakeLists.txt file. Also bear in mind that with this code the importer is already fully reconfigurable and instances of it can be created and removed at user’s leisure.

The effect

The effect

As always, you can check out my progress on my public Amarok clone. The branch is named gsoc-importers.

Thanks for reading!

GSoC Status Update – Week 9

This is a status update for my Google Summer of Code 2013 project – implementing advanced statistics importers for Amarok. Please read the first post if you would like to know more about the project.

Last week was free of surprises. I applied some more polish to both the Importers framework and concrete importers themselves, deduplicated some code – just general maintenance. I had a very technical to-do list containing – mostly – very minor entries, so it’s nothing write about in the post. The bottom line is: the overall quality of my project is improving.

The list keeps gaining new entries, so hopefully I’ll still be having plenty to do.

One thing that continues to give me a bit of trouble is Amarok 2.x embedded database importer. There are surprisingly many problems with managing a single server process, shared between method calls, in a multithreaded environment. QProcess class has its own set of quirks on top of that which need to be taken into account, especially when it comes to destroying the object, particularly in multithreaded environment. I’ve got ideas, but it’s something for the next post.

In other news: Banshee importer!

Everyone loves pictures

It still has some issues, but nothing that won’t be solved by this time next week

Alright, that’s me done for the post. It’s been a short one, but as I said – no major problems, no major changes. A steady march to high quality.

As always, you can check out my progress on my public Amarok clone. The branch is named gsoc-importers.

Thanks for reading!

GSoC Status Update – Week 8

This is a status update for my Google Summer of Code 2013 project – implementing advanced statistics importers for Amarok. Please read the first post if you would like to know more about the project.

My GSoC project is done!

So many plugins

Alright, not really. But, I already reached my hard goals and am well on the way to finishing up on soft goals as written in my proposal: tests and two-way synchronization. Reaching all the hard goals is something to celebrate!

Amarok 2.x importer

External connection config

External connection config

The Amarok 2.x importer is up and running, and it supports both kinds of Amarok database: external and embedded. The external one is pretty much your standard SQL-based importer. The embedded one is much more fun.

Embedded connection config

Embedded connection config

Obviously there’s no daemon running when dealing with embedded Amarok database; yet we have to start a MySQL server because data-mining its files is a particularly bad idea. The most straightforward approach would be to simply run an embedded MySQL server, but due to technical reasons we are limited to one embedded server per process. This means that if current Amarok process stores its collection in an embedded database – which is the default – we’re quite out of luck.

The solution I opted for is not perfect, but better than any other I could come up with: start a server as a sub-process. The child is fed delicious arguments, making it suitable to run in a multi-server environment, and under typical desktop user’s privileges. We then connect to the server and treat it in the same way as external one: that’s exactly what it is. The code is written in a way that starts the server on demand and shuts it down after a period of inactivity. Neat.

There are several drawbacks to this approach, the most obvious one being that mysqld binary has to be present on the system. A less obvious one, but which proved to be most troublesome, is lack of answer to a simple question: has the server started up yet? We can’t poll the output for equivalent of “ready to serve” message, as it can be written in any language… or can it? There may be another way to determine if the server is fully started, but I haven’t found one yet. So for now, after mysqld process starts, it’s given a small time window to get ready.

The future

In addition to things outlined in previous posts, I’ve got a few small tasks on my to-do list – just some overall edge-rounding and a little problem-solving. The process-handling part of Amarok 2.x importer will also be receiving a bit more love, and hopefully I’ll find a better solution to the problem mentioned above. I’ll also be looking into implementing Banshee importer.

As always, you can check out my progress on my public Amarok clone. The branch is named gsoc-importers.

Thanks for reading!