Konrad's weblog
2016-01-14T16:40:54+01:00
https://konradzemek.com
Konrad Zemek
konrad.zemek@gmail.com
Asio, SSL, and scalability
2015-08-16T15:06:00+02:00
https://konradzemek.com/2015/08/16/asio-ssl-and-scalability
<p>Let's say you want to build an SSL server in C++. Transmitting data is going to
be a major part of your application, so you need it to be fast and efficiently
use system resources, especially processor cores.</p>
<p>You may have heard of <a href="https://think-async.com">Asio</a> (possibly better known as <a href="http://www.boost.org/doc/libs/1_58_0/doc/html/boost_asio.html">Boost.Asio</a>). Asio is a
"cross-platform C++ library for network and low-level I/O programming that
provides developers with a consistent asynchronous model". It's widely used and
mature, and in the future <a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n4370.html">may be a part of C++17 standard library</a>. Note
that this post is not a tutorial or an introduction to Asio, but rather a study
on how scalable it is in our use case, why it scales poorly and how to improve
it.</p>
<h2 id="benchmarking">Benchmarking</h2>
<p>Asio includes SSL support using <a href="https://www.openssl.org">OpenSSL</a> library. I've created an <a href="https://gist.github.com/kzemek/9f7a8d94171197a9f88e">example,
"naive" benchmark</a> that sets up a server with a given number of
threads, creates a given number of connections and measures the time it takes
each of them to send <code>M</code> messages of size <code>N</code>. The code uses Asio 1.10.6, which
is the current stable version. Here's how I compile it on OS X:</p>
<div class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>clang++ -std<span class="o">=</span>c++11 -O3 benchmark_naive.cpp -o benchmark_naive <span class="se">\</span>
-Iasio-1.10.6/include/ -I/usr/local/opt/openssl/include/ <span class="se">\</span>
-L/usr/local/opt/openssl/lib/ -lssl -lcrypto
</code></pre></div>
<p>And here are example benchmark results on my machine (that has 4 cores with
Hyper-threading, adding up to 8 virtual cores):</p>
<div class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span><span class="c"># 1 thread, 1 connection, 1000 messages * 10 MB</span>
<span class="nv">$ </span>./benchmark_naive <span class="m">1</span> <span class="m">1</span> <span class="m">1000</span> <span class="k">$((</span><span class="m">10</span> <span class="o">*</span> <span class="m">1024</span> <span class="o">*</span> <span class="m">1024</span><span class="k">))</span>
<span class="m">10000</span> megabytes sent and received in 22.392 seconds. <span class="o">(</span>446.588 MB/s<span class="o">)</span>
<span class="nv">$ </span><span class="c"># 8 threads, 4 connections, 250 messages * 10 MB</span>
<span class="nv">$ </span>./benchmark_naive <span class="m">8</span> <span class="m">4</span> <span class="m">250</span> <span class="k">$((</span><span class="m">10</span> <span class="o">*</span> <span class="m">1024</span> <span class="o">*</span> <span class="m">1024</span><span class="k">))</span>
<span class="m">10000</span> megabytes sent and received in 52.821 seconds. <span class="o">(</span>189.319 MB/s<span class="o">)</span>
</code></pre></div>
<p>Let's put all of my results into a chart:</p>
<script>
window.addEventListener('load', function() {
var ctx = document.getElementById('asio-ssl-and-scalability_chart1').getContext('2d');
var data = {"labels":["1 thread","2 threads","3 threads","4 threads","5 threads","6 threads","7 threads","8 threads"],"datasets":[{"label":"1 connection","data":[459.074,632.431,675.63,672.179,668.852,672.405,673.355,670.691],"strokeColor":"#1f77b4","pointColor":"#1f77b4"},{"label":"4 connections","data":[457.206,723.17,447.107,205.415,196.04,194.989,192.864,193.013],"strokeColor":"#ff7f0e","pointColor":"#ff7f0e"},{"label":"10 connections","data":[457.624,723.327,424.106,207.065,196.052,192.404,193.633,194.382],"strokeColor":"#2ca02c","pointColor":"#2ca02c"}]};
var options = {"datasetFill":false,"multiTooltipTemplate":"<%=value%> MB/s"};
new Chart(ctx).Line(data, options);
legend(document.getElementById("asio-ssl-and-scalability_chart1_legend"), data);
}, false);
</script>
<canvas class="chart" id="asio-ssl-and-scalability_chart1"></canvas>
<div id="asio-ssl-and-scalability_chart1_legend"></div>
<p>As you can see, the chart shows an increase in bandwidth for two threads, which
we would expect even for a single connection (as one endpoint has to encrypt,
and other decrypt the data). After that point, <strong>the bandwidth rapidly falls off
with the number of threads</strong>, reaching about 200 MB/s when used with multiple
connections. Connections can potentially run in parallel, so the ideal chart
would show a linear speedup with the number of threads, capping at the ratio of
2 threads/connection.</p>
<p>Summarizing the experiment, Asio with OpenSSL not only does not scale with the
number of threads, it actually <em>slows down considerably</em> when the number of
concurrent operations is high enough. This suggests a heavy lock congestion.
Let's check our theory in a profiler:</p>
<!-- _includes/image.html -->
<figure class="image-wrapper">
<a href="/assets/2015/08/asio_profile.png" title="Profiling results" target="_blank">
<img src="/assets/2015/08/asio_profile.png" alt="Inverted call tree"/>
</a>
<figcaption class="image-caption">Inverted call tree</figcaption>
</figure>
<p>Most of the time of our application is spent waiting for locks, most of the
locking occurs in <code>ERR_get_state</code> OpenSSL function, and it can most often be
found in a call tree of <code>asio::ssl::detail::engine::perform</code>. A look at
<code>engine::perform</code> implementation in <a href="https://github.com/chriskohlhoff/asio/blob/asio-1-10-6/asio/include/asio/ssl/detail/impl/engine.ipp#L233">engine.ipp</a> quickly shows us that...
everything is fine. Sure, you can maybe make a few small optimizations, but in
the larger picture OpenSSL is used correctly<sup id="fnref1"><a href="#fn1" rel="footnote">1</a></sup><sup id="fnref2"><a href="#fn2" rel="footnote">2</a></sup>. There's no error there
that would result in the lock congestion. We conclude that the <strong>bottleneck in
scalability lies in OpenSSL itself</strong>, more specifically in its error handling
functions. So what now?</p>
<h2 id="trying-boringssl">Trying BoringSSL</h2>
<p>After <a href="https://en.wikipedia.org/wiki/Heartbleed">Heartbleed</a>, a few OpenSSL forks like <a href="http://www.libressl.org">LibreSSL</a> and <a href="https://boringssl.googlesource.com/boringssl">BoringSSL</a> have
appeared with a vision to trim down, modernize and secure OpenSSL's code while
remaining mostly source-compatible with the original. A quick look at
<a href="https://github.com/libressl/libressl/blob/ba04546060b835e279c0e8a26261e73a1964927c/src/crypto/err/err.c">LibreSSL's err.c</a> file shows it's largely unchanged from the <a href="https://github.com/openssl/openssl/blob/OpenSSL_1_0_2d/crypto/err/err.c">OpenSSL's
version</a>. But <a href="https://boringssl.googlesource.com/boringssl/+/74279b63428d9b25052207dd81121b67a847c20e/crypto/err/err.c">BoringSSL's err.c</a> has obviously been reworked
and now <strong>uses a thread-local storage instead of locking a global mutex for data
access</strong>!</p>
<p>As I mentioned, BoringSSL is <em>mostly</em> source-compatible with OpenSSL.
Unfortunately, there are <a href="https://gist.github.com/kzemek/37aa2a2138b2651f2c55">a few changes to make</a> before we
can use it with Asio. There are <a href="https://github.com/chriskohlhoff/asio/issues/52">open</a> <a href="https://github.com/chriskohlhoff/asio/issues/74">issues</a> in Asio
GitHub repository to integrate these changes into upstream, but even then
compatibility with BoringSSL is currently a moving target as the code is still
being cleaned up.</p>
<p>Let's compile the example with BoringSSL and run our benchmarks:</p>
<div class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>clang++ -std<span class="o">=</span>c++11 -O3 benchmark_naive.cpp -o benchmark_naive <span class="se">\</span>
-Iasio-1.10.6/include/ -Iboringssl/include/ <span class="se">\</span>
-Lboringssl/build/ssl/ -Lboringssl/build/crypto/ <span class="se">\</span>
-lssl -lcrypto
<span class="nv">$ </span><span class="c"># 8 threads, 4 connections, 250 messages * 10 MB</span>
<span class="nv">$ </span>./benchmark_naive <span class="m">8</span> <span class="m">4</span> <span class="m">250</span> <span class="k">$((</span><span class="m">10</span> <span class="o">*</span> <span class="m">1024</span> <span class="o">*</span> <span class="m">1024</span><span class="k">))</span>
<span class="m">10000</span> megabytes sent and received in 11.591 seconds. <span class="o">(</span>862.738 MB/s<span class="o">)</span>
</code></pre></div>
<script>
window.addEventListener('load', function() {
var ctx = document.getElementById('asio-ssl-and-scalability_chart2').getContext('2d');
var data = {"labels":["1 thread","2 threads","3 threads","4 threads","5 threads","6 threads","7 threads","8 threads"],"datasets":[{"label":"1 connection","data":[482.439,832.224,931.793,925.669,925.754,933.358,867.83,834.028],"strokeColor":"#1f77b4","pointColor":"#1f77b4"},{"label":"4 connections","data":[479.616,913.242,1229.86,1190.19,1124.86,942.596,891.663,864.902],"strokeColor":"#ff7f0e","pointColor":"#ff7f0e"},{"label":"10 connections","data":[504.566,937.998,1375.89,1332.98,1350.26,1280.41,1057.75,977.517],"strokeColor":"#2ca02c","pointColor":"#2ca02c"}]};
var options = {"datasetFill":false,"multiTooltipTemplate":"<%=value%> MB/s"};
new Chart(ctx).Line(data, options);
legend(document.getElementById("asio-ssl-and-scalability_chart2_legend"), data);
}, false);
</script>
<canvas class="chart" id="asio-ssl-and-scalability_chart2"></canvas>
<div id="asio-ssl-and-scalability_chart2_legend"></div>
<p>Well, that's <em>much better</em>. You may end reading here, knowing that as of this
time OpenSSL's error handling causes it to scale poorly with the number of
threads, while <strong>BoringSSL scales much better indeed</strong>. But there's still a
fall-off in bandwidth when the number of threads increases, so let's try to
answer that as well.</p>
<h2 id="cores-threads-and-io_services">Cores, threads and io_services</h2>
<p>If you want to find a bottleneck, profiling is usually the best answer:</p>
<!-- _includes/image.html -->
<figure class="image-wrapper">
<a href="/assets/2015/08/asio_boringssl_profile.png" title="Profiling results with BoringSSL" target="_blank">
<img src="/assets/2015/08/asio_boringssl_profile.png" alt="Inverted call tree with BoringSSL"/>
</a>
<figcaption class="image-caption">Inverted call tree with BoringSSL</figcaption>
</figure>
<p>Looks like Asio's internal thread synchronization is the bottleneck this time.
There are two main approaches to get scalability in Asio: <code>thread-per-core</code> and
<code>io_service-per-core</code>. In the "naive" example I'm using the <code>thread-per-core</code>
approach as it always seemed more natural to me - we're basically creating a
threadpool that - if needed - can dedicate one or more threads to a single
connection, as opposed to <code>io_service-per-core</code> where each connection would be
served by at most one thread. But in the light of our profiling results I've
<a href="https://gist.github.com/kzemek/166e2af5f799d4f833a3">modified the example to use the second approach</a>.</p>
<p>I've added a new class, <code>IoServices</code>, objects of which hold multiple
<code>io_service</code>s. When we need an <code>io_service</code> - e.g. for a new client- or
server-side socket - we call <code>ioServices.get()</code> which will return one of the
stored <code>io_service</code> objects on a round-robin basis.</p>
<p>Let's put the results into a final chart. This time we'll just focus on 8
threads, 20 connections, and directly compare our different benchmark
applications. Note that I'm cheating here, if just a little bit:
<code>io_service-per-core</code> will perform better when there are multiple connections
per <code>io_service</code>, as it will result in a more balanced CPU load. Since we're
considering a server scenario, though, 20 connections is still a very low
number.</p>
<script>
window.addEventListener('load', function() {
var ctx = document.getElementById('asio-ssl-and-scalability_chart3').getContext('2d');
var data = {"labels":["1 thread","2 threads","3 threads","4 threads","5 threads","6 threads","7 threads","8 threads"],"datasets":[{"label":"thread-per-core w/ OpenSSL","data":[443.012,691.563,422.498,193.02,190.523,185.653,189.007,191.928],"strokeColor":"#1f77b4","pointColor":"#1f77b4"},{"label":"thread-per-core","data":[480.152,926.183,1288.33,1221.75,1192.29,1097.48,890.274,875.254],"strokeColor":"#ff7f0e","pointColor":"#ff7f0e"},{"label":"io_service-per-core","data":[493.772,779.195,1286.59,1375.71,1502.46,1687.27,1821.33,1620.35],"strokeColor":"#2ca02c","pointColor":"#2ca02c"}]};
var options = {"datasetFill":false,"multiTooltipTemplate":"<%=value%> MB/s"};
new Chart(ctx).Line(data, options);
legend(document.getElementById("asio-ssl-and-scalability_chart3_legend"), data);
}, false);
</script>
<canvas class="chart" id="asio-ssl-and-scalability_chart3"></canvas>
<div id="asio-ssl-and-scalability_chart3_legend"></div>
<p>This still isn't the ideal chart (of course, due to threads' overhead no
implementation can exist that would produce the ideal results), but it actually
<strong>scales up with the number of used threads</strong>.</p>
<p>That's it for the post. Now that you know how to make Asio-based SSL server
scale, you can go build a faster and safer applications.</p>
<p>Thanks for reading!</p>
<div class="footnotes">
<hr>
<ol>
<li id="fn1">
<p><strong>UPDATE 2015-08-17</strong>: While it's true that Asio uses OpenSSL API
correctly, it's possible to write multi-threaded code that uses OpenSSL
and which is not constrained by the error-handling bottleneck. Asio calls
<code>ERR_clear_error()</code> before each call to <code>SSL_*()</code> functions, as OpenSSL
documentation states <em>"The current thread's error queue must be empty
before the TLS/SSL I/O operation is attempted"</em>. To avoid the bottleneck,
instead of clearing the error queue before each operation, you have to
make <em>sure</em> to clear the queue after an error has occurred. This is
something that can be done in Asio code. <a href="#fnref1" rev="footnote">↩</a></p>
</li>
<li id="fn2">
<p><strong>UPDATE 2015-08-20</strong>: The update above turned out to be incorrect, as
OpenSSL will still call locking functions internally. While the solution I
described above will decrease lock contention, it doesn't completely
remove the bottleneck. <a href="#fnref2" rev="footnote">↩</a></p>
</li>
</ol>
</div>
The end of GSoC 2013
2013-09-23T21:04:00+02:00
https://konradzemek.com/2013/09/23/the-end-of-gsoc-2013
<blockquote>
<p>This is a status update for my <a href="https://www.google-melange.com/gsoc/homepage/google/gsoc2013">Google Summer of Code 2013</a> project -
implementing advanced statistics importers for <a href="https://amarok.kde.org">Amarok</a>. Please read <a href="/2013/06/15/hello-world-me-the-blog-and-gsoc/">the
first post</a> if you would like to know more about the project.</p>
</blockquote>
<p>So here came the end of Google Summer of Code 2013. It's the best moment to talk
about what has been planned, what was done, and - traditionally - what the
future holds. Oh, and let's get some stats and pictures (because that's what
we're here for)!</p>
<h2 id="plans">Plans</h2>
<p>I <a href="/2013/06/15/hello-world-me-the-blog-and-gsoc/">planned</a> to:</p>
<ul>
<li>rewrite following statistics importers based on the existing StatSyncing
framework:
<ul>
<li>Amarok 1.4 (FastForward)</li>
<li>Apple iTunes</li>
</ul></li>
<li>create new statistics importers for the following media players:
<ul>
<li>Amarok 2.x</li>
<li>Rhythmbox</li>
</ul></li>
<li><strong>soft goal</strong> two-way synchronization for created importers (I guess I
shouldn't call them "importers" anymore, but something like "synchronization
targets" just sounds too unwieldy)</li>
<li><strong>softer goal</strong> write tests for created importers</li>
<li><strong>side goal</strong> make new importers easy to write.</li>
</ul>
<h2 id="reality">Reality</h2>
<p>We all know that thing about plans and reality in software development, right?</p>
<!-- _includes/image.html -->
<figure class="image-wrapper">
<a href="/assets/2013/09/tree_swing.jpg" title="Development tree swing" target="_blank">
<img src="/assets/2013/09/tree_swing.jpg" alt="©2005 Paragon Innovations, Inc. I promised you pictures."/>
</a>
<figcaption class="image-caption">©2005 Paragon Innovations, Inc. I promised you pictures.</figcaption>
</figure>
<p>So let's recap what I've done:</p>
<ul>
<li>I created a framework for importers that takes care of managing them, loading
and saving their configuration
<ul>
<li>the consequence is that a user can create, remove, reconfigure new importers
on a whim</li>
<li>programmer can also make as many configuration options as he wants and store
as much state in the importer as needed</li>
</ul></li>
<li>I created following statistics importer <em>types</em>, loadable as plugins:
<ul>
<li>Amarok 1.4</li>
<li>Amarok 2.4</li>
<li>Apple iTunes</li>
<li>Banshee</li>
<li>Clementine</li>
<li>Rhythmbox</li>
</ul></li>
<li>...and I implemented two-way synchronization for each of them</li>
<li>I wrote tests for every importer and all parts of the framework (except for
one helper, as it's destined to be removed)</li>
<li>I made new importers very easy to create - and they automatically gain all
goodies from the framework.</li>
</ul>
<p>Please take a look at earlier posts if you're interested in pictures - I didn't
want to overload this one with duplicates.</p>
<h2 id="statistics">Statistics</h2>
<p>Let's get the total number of changes reported by git:</p>
<div class="highlight"><pre><code class="language-sh" data-lang="sh">konrad@KonPC-Linux ~/kde/src/amarok <span class="nv">$ </span>git log a214c0a2^..HEAD --shortstat <span class="se">\</span>
<span class="p">|</span> grep -E <span class="s2">"file(s)? changed"</span> <span class="se">\</span>
<span class="p">|</span> awk <span class="s1">'{inserted+=$4; deleted+=$6} END \</span>
<span class="s1"> {print "lines inserted:", inserted, "lines deleted:", deleted}'</span>
lines inserted: <span class="m">21529</span> lines deleted: 10225
</code></pre></div>
<p>That's a nice number of changes. Here's what it actually sums up to:</p>
<div class="highlight"><pre><code class="language-sh" data-lang="sh">konrad@KonPC-Linux ~/kde/src/amarok <span class="nv">$ </span>git diff a214c0a2^ HEAD --shortstat
<span class="m">303</span> files changed, <span class="m">13519</span> insertions<span class="o">(</span>+<span class="o">)</span>, <span class="m">2257</span> deletions<span class="o">(</span>-<span class="o">)</span>
</code></pre></div>
<p>The difference between these numbers tells us a story: at least about ⅖ of the
code I wrote was removed afterwards through <strong>heavy refactoring</strong>. (Alright,
maybe this wasn't much of a story.) In other words I not only overshot my
initial plans, I was also making sure I'm doing it with <em>style</em>. ;) Of course
the numbers take into account only the changes that were finally committed -
there were a lot more in between.</p>
<p>Let's make some code size comparisons between old and new importers.</p>
<!-- _includes/image.html -->
<figure class="image-wrapper">
<a href="/assets/2013/09/codeSize.png" title="Code size graph" target="_blank">
<img src="/assets/2013/09/codeSize.png" alt="I really wanted some graphs in this post."/>
</a>
<figcaption class="image-caption">I really wanted some graphs in this post.</figcaption>
</figure>
<p>As you can see, the numbers are similar. Rhythmbox and iTunes importers are made
bigger by XML-processing code (oh how I hate it), and Amarok 2.x and FastForward
importers by custom, rich configuration widgets. The simplest importers,
Clementine and Banshee, are small and pretty.</p>
<p>Oh, but that's not the whole story here, is it? All of the new importers also
contain write capabilities - they can sync the statistics back to the foreign
media player. Without it a new importer can easily fit inside 100 lines, as
demonstrated in <a href="/2013/08/27/gsoc-status-update-week-10-2/">one of my previous posts</a>. Mission accomplished.</p>
<p>As an aside, I find it interesting to note that the number of lines does
translate very well into the number of characters. The average number of
characters per line for measured code is 38.17, with standard deviation of 2.60
characters between files.</p>
<p>Oh, and I have proof that I was actually <em>doing</em> something during GSoC. Do take
a look at this video that I made, if only for the amazing soundtrack (720p
recommended). For details, please see the <a href="https://youtu.be/Cj6zmDUJtaE">video's description</a>.</p>
<div class="youtube">
<div>
<iframe src="//www.youtube-nocookie.com/embed/Cj6zmDUJtaE?rel=0" allowfullscreen="allowfullscreen" frameborder="0"></iframe>
</div>
</div>
<h2 id="the-future">The future</h2>
<p>So, it's the end of Google Summer of Code, but it's not the end of the project
nor my contribution to Amarok. Arguably the most important event in the
project's lifetime - code review - still lies ahead. Not only that, but I
already have some further refactorings in mind.</p>
<p>Other than the project, there's always a lot to do for Amarok, and the great
community around it makes it hard to leave - so I'm not. There's just too much
fun to be had. ;)</p>
<p>Well, that's it for the post. I'm going to take a few days off and then an
academic term starts, and I'm back to my daytime job - I'll need some time to
adjust my schedule. Thanks for sticking around. It's been - and continues to be
- a pleasure!</p>
GSoC Status Update – Week 11 & 12
2013-09-09T18:08:00+02:00
https://konradzemek.com/2013/09/09/gsoc-status-update-week-11-12
<blockquote>
<p>This is a status update for my <a href="https://www.google-melange.com/gsoc/homepage/google/gsoc2013">Google Summer of Code 2013</a> project -
implementing advanced statistics importers for <a href="https://amarok.kde.org">Amarok</a>. Please read <a href="/2013/06/15/hello-world-me-the-blog-and-gsoc/">the
first post</a> if you would like to know more about the project.</p>
</blockquote>
<p>Last week I asked my mentor if I could skip that week's report and make the next
one (i.e. this one) a double one instead. The reason for this was that I was
working almost exclusively on tests, and tests more often than not make for a
dull post.</p>
<p>Well, after two weeks I have even more tests. I'd go as far as to say that
things are satisfyingly tested.</p>
<p>I made a base test suite that relies on convention; importers are expected to
have tracks with certain metadata in their databases, and then tests make sure
that this data is imported correctly. Tracks with the right metadata are pre-
created and stored in the source tree so creating a test database is a matter of
adding them to media player's collection.</p>
<p>There isn't much more to tests that's interesting, so let's skip to the next
topic. <strong>GSoC 2013 is in the homestretch.</strong> September 16 is "soft pencils-down"
and September 23 a "hard pencils-down" date. We're expected to have all the code
done on September 16, and to spend the following week on documentation. Having
that in mind, this is my much-more-detailed-than-usual plan for this week:</p>
<ol>
<li><strong>Monday</strong> Amarok 2.x importer two-way synchronization <em>(done)</em></li>
<li><strong>Tuesday</strong> FastForward importer two-way synchronization</li>
<li><strong>Wednesday</strong> Banshee & Clementine importers two-way synchronization</li>
<li><strong>Thursday</strong> Rhythmbox importer two-way synchronization</li>
<li><strong>Friday</strong> iTunes importer two-way synchronization<sup id="fnref1"><a href="#fn1" rel="footnote">1</a></sup></li>
<li><strong>weekend</strong> more tests! Supplement existing test suites.</li>
</ol>
<p>Since I've been documenting as I went, the week between September 16 and 23 will
be devoted to small bugfixes, design tweaks, typo fixes. Also that's the week
where I remove old media-player importing capabilities and take a minute or two
to celebrate.</p>
<p>As always, you can check out my progress on my <a href="https://quickgit.kde.org/?p=clones%2Famarok%2Fkzemek%2Famarok.git">public Amarok clone</a>. The branch
is named <code>gsoc-importers</code>.</p>
<p>Thanks for reading!</p>
<div class="footnotes">
<hr>
<ol>
<li id="fn1">
<p>iTunes database will then have to be imported back into iTunes.
But hey, it's synchronization. Kind of. <a href="#fnref1" rev="footnote">↩</a></p>
</li>
</ol>
</div>
GSoC Status Update – Week 10
2013-08-27T18:20:00+02:00
https://konradzemek.com/2013/08/27/gsoc-status-update-week-10-2
<blockquote>
<p>This is a status update for my <a href="https://www.google-melange.com/gsoc/homepage/google/gsoc2013">Google Summer of Code 2013</a> project -
implementing advanced statistics importers for <a href="https://amarok.kde.org">Amarok</a>. Please read <a href="/2013/06/15/hello-world-me-the-blog-and-gsoc/">the
first post</a> if you would like to know more about the project.</p>
</blockquote>
<h2 id="fixing">Fixing</h2>
<p>Last week I fixed Amarok 2.x embedded database importer. There were quite a few
problems with handling an external database process:</p>
<ul>
<li>if the <code>QProcess</code> object received commands (<code>start()</code>, <code>kill()</code>,
<code>terminate()</code>) from a different thread than that which has created it, it
resulted in wrong behavior, often in the form of a crash (can't get much
wronger than <em>that</em>)</li>
<li>the <code>QProcess</code> object does <strong>not</strong> encapsulate the process; it's only an
interface. When a <code>QProcess</code> object was destroyed while the process was still
running, it issued a warning and killed the process with SIGKILL. Normally
you'd want to stop an ongoing process with a SIGTERM, so manual lifetime
management is needed</li>
<li>if the server has stopped (which it does, after a period of inactivity),
calling <code>QSqlDatabase::removeDatabase()</code> resulted in a SIGPIPE signal from
inside the MySQL driver</li>
<li>related to that, an old <code>QSqlDatabase</code> connection silently failed to work if
the server has been restarted. There would be no warnings on
<code>QSqlDatabase::open()</code>.</li>
</ul>
<p>There was also a question of waiting for mysqld process to be ready to serve.
After some research I decided to adopt an approach that MySQL startup scripts
(including the "official" script, mysql.server) use, which is to wait for the
server's PID file to be modified. Overall, I'm quite satisfied with the results;
it's reliable and fast enough.</p>
<h2 id="creating">Creating</h2>
<p>By the way, there's a new statistics synchronization target: Clementine.</p>
<!-- _includes/image.html -->
<figure class="image-wrapper">
<a href="/assets/2013/08/clementine-importer.png" title="Importers window" target="_blank">
<img src="/assets/2013/08/clementine-importer.png" alt="I know how much you like pictures"/>
</a>
<figcaption class="image-caption">I know how much you like pictures</figcaption>
</figure>
<p>This brings the total number of importers to six, and marks the end of
implementing new importers. From now on, I intend to focus on existing code
including, but not limited to, read-write capabilities and - of course -
testing.</p>
<h2 id="simplifying">Simplifying</h2>
<p>You may have noticed one additional target on the screenshot above: "Example."
Another thing I focused on this week was code deduplication and simplifying
creation of new importers. To show off, I prepared a basic "Example" importer.
Below is the <em>full</em> C++ code. Bear in mind that aside from the code, importers
need a plugin's *.desktop file and a CMakeLists.txt file. Also bear in mind that
with this code the importer is already fully reconfigurable and instances of it
can be created and removed at user's leisure.</p>
<div class="highlight"><pre><code class="language-c++" data-lang="c++"><span class="cp">#include "importers/SimpleImporterConfigWidget.h"</span>
<span class="cp">#include "importers/SimpleImporterManager.h"</span>
<span class="cp">#include "importers/ImporterProvider.h"</span>
<span class="cp">#include "statsyncing/SimpleTrack.h"</span>
<span class="k">using</span> <span class="k">namespace</span> <span class="n">StatSyncing</span><span class="p">;</span>
<span class="k">class</span> <span class="nc">ExampleConfigWidget</span> <span class="o">:</span> <span class="k">public</span> <span class="n">SimpleImporterConfigWidget</span>
<span class="p">{</span>
<span class="k">public</span><span class="o">:</span>
<span class="k">explicit</span> <span class="n">ExampleConfigWidget</span><span class="p">(</span> <span class="k">const</span> <span class="n">QVariantMap</span> <span class="o">&</span><span class="n">config</span><span class="p">,</span> <span class="n">QWidget</span> <span class="o">*</span><span class="n">parent</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="n">Qt</span><span class="o">::</span><span class="n">WindowFlags</span> <span class="n">f</span> <span class="o">=</span> <span class="mi">0</span> <span class="p">)</span>
<span class="o">:</span> <span class="n">SimpleImporterConfigWidget</span><span class="p">(</span> <span class="s">"Example"</span><span class="p">,</span> <span class="n">config</span><span class="p">,</span> <span class="n">parent</span><span class="p">,</span> <span class="n">f</span> <span class="p">)</span> <span class="p">{}</span>
<span class="p">};</span>
<span class="k">class</span> <span class="nc">ExampleProvider</span> <span class="o">:</span> <span class="k">public</span> <span class="n">ImporterProvider</span>
<span class="p">{</span>
<span class="k">public</span><span class="o">:</span>
<span class="n">ExampleProvider</span><span class="p">(</span> <span class="k">const</span> <span class="n">QVariantMap</span> <span class="o">&</span><span class="n">config</span><span class="p">,</span> <span class="n">ImporterManager</span> <span class="o">*</span><span class="n">manager</span> <span class="p">)</span>
<span class="o">:</span> <span class="n">ImporterProvider</span><span class="p">(</span> <span class="n">config</span><span class="p">,</span> <span class="n">manager</span> <span class="p">)</span> <span class="p">{}</span>
<span class="n">qint64</span> <span class="n">reliableTrackMetaData</span><span class="p">()</span> <span class="k">const</span> <span class="p">{</span> <span class="k">return</span> <span class="n">Meta</span><span class="o">::</span><span class="n">valTitle</span> <span class="o">|</span> <span class="n">Meta</span><span class="o">::</span><span class="n">valArtist</span> <span class="o">|</span> <span class="n">Meta</span><span class="o">::</span><span class="n">valAlbum</span><span class="p">;</span> <span class="p">}</span>
<span class="n">qint64</span> <span class="n">writableTrackStatsData</span><span class="p">()</span> <span class="k">const</span> <span class="p">{</span> <span class="k">return</span> <span class="mi">0</span><span class="p">;</span> <span class="p">}</span>
<span class="n">QSet</span><span class="o"><</span><span class="n">QString</span><span class="o">></span> <span class="n">artists</span><span class="p">()</span> <span class="p">{</span> <span class="k">return</span> <span class="n">QSet</span><span class="o"><</span><span class="n">QString</span><span class="o">></span><span class="p">()</span> <span class="o"><<</span> <span class="s">"ExampleArtist"</span><span class="p">;</span> <span class="p">}</span>
<span class="n">TrackList</span> <span class="n">artistTracks</span><span class="p">(</span> <span class="k">const</span> <span class="n">QString</span> <span class="o">&</span><span class="n">artistName</span> <span class="p">)</span>
<span class="p">{</span>
<span class="n">Meta</span><span class="o">::</span><span class="n">FieldHash</span> <span class="n">metadata</span><span class="p">;</span>
<span class="n">metadata</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span> <span class="n">Meta</span><span class="o">::</span><span class="n">valArtist</span><span class="p">,</span> <span class="n">artistName</span> <span class="p">);</span>
<span class="n">metadata</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span> <span class="n">Meta</span><span class="o">::</span><span class="n">valTitle</span><span class="p">,</span> <span class="s">"ExampleTitle"</span> <span class="p">);</span>
<span class="n">metadata</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span> <span class="n">Meta</span><span class="o">::</span><span class="n">valAlbum</span><span class="p">,</span> <span class="s">"ExampleAlbum"</span> <span class="p">);</span>
<span class="n">metadata</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span> <span class="n">Meta</span><span class="o">::</span><span class="n">valRating</span><span class="p">,</span> <span class="mi">10</span> <span class="p">);</span>
<span class="k">return</span> <span class="nf">TrackList</span><span class="p">()</span> <span class="o"><<</span> <span class="n">TrackPtr</span><span class="p">(</span> <span class="k">new</span> <span class="n">SimpleTrack</span><span class="p">(</span> <span class="n">metadata</span> <span class="p">)</span> <span class="p">);</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="n">AMAROK_EXPORT_SIMPLE_IMPORTER_PLUGIN</span><span class="p">(</span> <span class="n">example</span><span class="p">,</span> <span class="s">"ExampleImporter"</span><span class="p">,</span> <span class="n">i18n</span><span class="p">(</span> <span class="s">"Example"</span> <span class="p">),</span>
<span class="n">i18n</span><span class="p">(</span> <span class="s">"Example Statistics Importer"</span> <span class="p">),</span> <span class="n">KIcon</span><span class="p">(</span> <span class="s">"dialog-ok"</span> <span class="p">),</span>
<span class="n">ExampleConfigWidget</span><span class="p">,</span> <span class="n">ExampleProvider</span> <span class="p">)</span>
</code></pre></div>
<!-- _includes/image.html -->
<figure class="image-wrapper">
<a href="/assets/2013/08/example-importer.png" title="Creted importer" target="_blank">
<img src="/assets/2013/08/example-importer.png" alt="The effect"/>
</a>
<figcaption class="image-caption">The effect</figcaption>
</figure>
<p>As always, you can check out my progress on my [public Amarok clone. The branch
is named <code>gsoc-importers</code>.</p>
<p>Thanks for reading!</p>
GSoC Status Update – Week 9
2013-08-19T22:52:00+02:00
https://konradzemek.com/2013/08/19/gsoc-status-update-week-9
<blockquote>
<p>This is a status update for my <a href="https://www.google-melange.com/gsoc/homepage/google/gsoc2013">Google Summer of Code 2013</a> project -
implementing advanced statistics importers for <a href="https://amarok.kde.org">Amarok</a>. Please read <a href="/2013/06/15/hello-world-me-the-blog-and-gsoc/">the
first post</a> if you would like to know more about the project.</p>
</blockquote>
<p>Last week was free of surprises. I applied some more polish to both the
Importers framework and concrete importers themselves, deduplicated some code -
just general maintenance. I had a very technical to-do list containing - mostly
- very minor entries, so it's nothing write about in the post. The bottom line
is: the overall quality of my project is improving.<</p>
<p>The list keeps gaining new entries, so hopefully I'll still be having plenty to
do.</p>
<p>One thing that continues to give me a bit of trouble is Amarok 2.x embedded
database importer. There are surprisingly many problems with managing a single
server process, shared between method calls, in a multithreaded environment.
QProcess class has its own set of quirks on top of that which need to be taken
into account, especially when it comes to destroying the object, <em>particularly</em>
in multithreaded environment. I've got ideas, but it's something for the next
post.</p>
<p>In other news: Banshee importer!</p>
<!-- _includes/image.html -->
<figure class="image-wrapper">
<a href="/assets/2013/08/banshee-importer.png" title="Importers window" target="_blank">
<img src="/assets/2013/08/banshee-importer.png" alt="It still has some issues, but nothing that won't be solved by this time next week"/>
</a>
<figcaption class="image-caption">It still has some issues, but nothing that won't be solved by this time next week</figcaption>
</figure>
<p>Alright, that's me done for the post. It's been a short one, but as I said - no
major problems, no major changes. A steady march to high quality.</p>
<p>As always, you can check out my progress on my <a href="https://quickgit.kde.org/?p=clones%2Famarok%2Fkzemek%2Famarok.git">public Amarok clone</a>. The branch
is named <code>gsoc-importers</code>.</p>
<p>Thanks for reading!</p>
GSoC Status Update – Week 8
2013-08-13T21:06:00+02:00
https://konradzemek.com/2013/08/13/gsoc-status-update-week-8
<blockquote>
<p>This is a status update for my <a href="https://www.google-melange.com/gsoc/homepage/google/gsoc2013">Google Summer of Code 2013</a> project -
implementing advanced statistics importers for <a href="https://amarok.kde.org">Amarok</a>. Please read <a href="/2013/06/15/hello-world-me-the-blog-and-gsoc/">the
first post</a> if you would like to know more about the project.</p>
</blockquote>
<h2 id="my-gsoc-project-is-done">My GSoC project is done!</h2>
<!-- _includes/image.html -->
<figure class="image-wrapper">
<a href="/assets/2013/08/plugins_amarok_2x.png" title="Plugins list" target="_blank">
<img src="/assets/2013/08/plugins_amarok_2x.png" alt="So many plugins!"/>
</a>
<figcaption class="image-caption">So many plugins!</figcaption>
</figure>
<p>Alright, not really. <strong>But</strong>, I already reached my hard goals and am well on the
way to finishing up on soft goals as written in my proposal: tests and two-way
synchronization. Reaching all the hard goals is something to celebrate!</p>
<h2 id="amarok-2-x-importer">Amarok 2.x importer</h2>
<!-- _includes/image.html -->
<figure class="image-wrapper">
<a href="/assets/2013/08/add_amarok_2x_external.png" title="" target="_blank">
<img src="/assets/2013/08/add_amarok_2x_external.png" alt="External connection config"/>
</a>
<figcaption class="image-caption">External connection config</figcaption>
</figure>
<p>The Amarok 2.x importer is up and running, and it supports both kinds of Amarok
database: external and embedded. The external one is pretty much your standard
SQL-based importer. The embedded one is <em>much more fun</em>.</p>
<!-- _includes/image.html -->
<figure class="image-wrapper">
<a href="/assets/2013/08/add_amarok_2x_embedded.png" title="" target="_blank">
<img src="/assets/2013/08/add_amarok_2x_embedded.png" alt="Embedded connection config"/>
</a>
<figcaption class="image-caption">Embedded connection config</figcaption>
</figure>
<p>Obviously there's no daemon running when dealing with embedded Amarok database;
yet we <em>have</em> to start a MySQL server because data-mining its files is a
particularly bad idea. The most straightforward approach would be to simply run
an embedded MySQL server, but due to technical reasons we are limited to one
embedded server per process. This means that if current Amarok process stores
its collection in an embedded database - which is the default - we're quite out
of luck.</p>
<p>The solution I opted for is not perfect, but better than any other I could come
up with: start a server as a sub-process. The child is fed delicious arguments,
making it suitable to run in a multi-server environment, and under typical
desktop user's privileges. We then connect to the server and treat it in the
same way as external one: that's exactly what it is. The code is written in a
way that starts the server on demand and shuts it down after a period of
inactivity. Neat.</p>
<p>There are several drawbacks to this approach, the most obvious one being that
mysqld binary has to be present on the system. A less obvious one, but which
proved to be most troublesome, is lack of answer to a simple question: has the
server started up yet? We can't poll the output for equivalent of "ready to
serve" message, as it can be written in any language... or can it? There <em>may</em>
be another way to determine if the server is fully started, but I haven't found
one yet. So for now, after mysqld process starts, it's given a small time window
to get ready.</p>
<h2 id="the-future">The future</h2>
<p>In addition to things outlined in previous posts, I've got a few small tasks on
my to-do list - just some overall edge-rounding and a little problem-solving.
The process-handling part of Amarok 2.x importer will also be receiving a bit
more love, and hopefully I'll find a better solution to the problem mentioned
above. I'll also be looking into implementing <strong>Banshee importer</strong>.</p>
<p>As always, you can check out my progress on my <a href="https://quickgit.kde.org/?p=clones%2Famarok%2Fkzemek%2Famarok.git">public Amarok clone</a>. The branch
is named <code>gsoc-importers</code>.</p>
<p>Thanks for reading!</p>
GSoC Status Update - Week 7
2013-08-05T20:17:00+02:00
https://konradzemek.com/2013/08/05/gsoc-status-update-week-7
<blockquote>
<p>This is a status update for my <a href="https://www.google-melange.com/gsoc/homepage/google/gsoc2013">Google Summer of Code 2013</a> project -
implementing advanced statistics importers for <a href="https://amarok.kde.org">Amarok</a>. Please read <a href="/2013/06/15/hello-world-me-the-blog-and-gsoc/">the
first post</a> if you would like to know more about the project.</p>
</blockquote>
<p>Well, this is going to be kind of a boring update. I implemented unit tests for
everything, Importers infrastructure (checking the documented properties,
testing interactions) and concrete importers alike. There were some minor design
changes but nothing interesting to talk about.</p>
<!-- _includes/image.html -->
<figure class="image-wrapper">
<a href="/assets/2013/08/testresults.png" title="Test results in console" target="_blank">
<img src="/assets/2013/08/testresults.png" alt="Who doesn't like to look at successful test results?"/>
</a>
<figcaption class="image-caption">Who doesn't like to look at successful test results?</figcaption>
</figure>
<p>While making tests, I found and fixed a couple of small issues. So, nothing out
of ordinary, stuff simply <em>works</em>. I don't feel that the tests are necessarily
complete (and frankly, no tests are <em>ever</em> complete), but I'll still be working
on them and adding new ones when I see fit. It's still barely past the GSoC
midterm!</p>
<p>Oh, yeah, these tests I wrote at the beginning of the project? Not in the
current branch. These were acceptance tests for synchronization - with new
importers, it's the StatSyncing framework's responsibility and I realized that
this kind of testing specifically for importers has no sense. So, these tests
may yet see a come-back, in one form or another, but not as a part of Importers'
test suite - rather as more generic StatSyncing framework tests. It's a low
priority thing, though.</p>
<p>As always, you can check out my progress on my <a href="https://quickgit.kde.org/?p=clones%2Famarok%2Fkzemek%2Famarok.git">public Amarok clone</a>. The branch
is named <code>gsoc-importers</code>.</p>
<p>Thanks for reading! <strong>Coming up next week - Amarok 2.x synchronization!</strong></p>
GSoC Status Update – Week 6
2013-07-29T21:29:00+02:00
https://konradzemek.com/2013/07/29/gsoc-status-update-week-6
<blockquote>
<p>This is a status update for my <a href="https://www.google-melange.com/gsoc/homepage/google/gsoc2013">Google Summer of Code 2013</a> project -
implementing advanced statistics importers for <a href="https://amarok.kde.org">Amarok</a>. Please read <a href="/2013/06/15/hello-world-me-the-blog-and-gsoc/">the
first post</a> if you would like to know more about the project.</p>
</blockquote>
<p>This week, I was meant to implement iTunes importer, bringing the total of
created importers to two.</p>
<!-- _includes/gallery.html -->
<div class="gallery">
<div>
<figure class="figure-2">
<a href="/assets/2013/07/ff_itunes_rhythmbox.png" target="_blank">
<img src="/assets/2013/07/ff_itunes_rhythmbox.png" alt=""/>
</a>
</figure>
</a>
<figure class="figure-2 figure-margin">
<a href="/assets/2013/07/ff_itunes_rhythmbox1.png" target="_blank">
<img src="/assets/2013/07/ff_itunes_rhythmbox1.png" alt=""/>
</a>
</figure>
</a>
</div>
<div>
<figure class="figure-2">
<a href="/assets/2013/07/ff_itunes_rhythmbox2.png" target="_blank">
<img src="/assets/2013/07/ff_itunes_rhythmbox2.png" alt=""/>
</a>
</figure>
</a>
<figure class="figure-2 figure-margin">
<a href="/assets/2013/07/ff_itunes_rhythmbox3.png" target="_blank">
<img src="/assets/2013/07/ff_itunes_rhythmbox3.png" alt=""/>
</a>
</figure>
</a>
</div>
</div>
<p>So, as you can see, I kind of overshot my goal here.</p>
<p>What happened is, importers are now really easy and fast to implement. The only
really piece of programming required is to retrieve track data from whatever
source (in case of both iTunes and Rhythmbox the source is an xml file), and all
the rest is just filling the blanks.</p>
<p>As is visible on the screenshots, the icon used for iTunes is a generic double
eight note used elsewhere in Amarok as well - it will probably stay that way.
The fact is, Apple does not grant a blanket license for usage of their logos,
and different countries have different notions of what consists of "fair use" of
such an icon. So there's that.</p>
<p>I also managed to implement a prototype (i.e. it's almost done) ID3v2 importer,
reading POPM (Popularimeter) frame for statistics and otherwise acting as a
wrapper for existing collection's track. I'm conflicted about its usefulness,
though, as Amarok already reads POPM on collection rescan provided that
corresponding <a href="http://gitorious.org/xdg-specs/xdg-specs/blobs/master/specifications/FMPSpecs/specification.txt%22">FMPS tags</a> are not set - if they are, they have precedence.
Besides initial importing, which should be doable by simply removing FMPS tags
and doing a full rescan, there's also question of updating these statistics, but
I've yet to find a prominent media player using POPM as a primary way of storing
ratings and playcounts. And for a good reason - to begin with, there's no
standard as to what different ratings mean (the values range from 1 to 255), and
POPM is an ID3v2 feature only - not that useful from media player's point of
view.</p>
<p>So, the future is now. Some more importer targets can definitely be expected
(Clementine? Foobar2000? I guess I'm taking requests!). Aside from that, I'm
still not done yet. Here's a short, by no means comprehensive list of what I
have yet to address:</p>
<ul>
<li><strong>code deduplication:</strong> right now declarations for importers' classes alone
can easily be longer than actual logic,</li>
<li><strong>Amarok 2.x importer</strong>,</li>
<li><strong>robustness, error handling:</strong> if anything, all plausible errors should cause
a warning in the terminal (and definitely <em>nothing</em> should break),</li>
<li><strong>tests:</strong> this can take some time to adapt to the new framework,</li>
<li><strong>two-way synchronization:</strong> this may very well introduce its own set of
problems, with synchronization and whatnot.</li>
</ul>
<p>As always, you can check out my progress on my <a href="https://quickgit.kde.org/?p=clones%2Famarok%2Fkzemek%2Famarok.git">public Amarok clone</a>. The branch
is named <code>gsoc-importers</code>.</p>
<p>Thanks for reading!</p>
GSoC Status Update - Week 5
2013-07-22T12:06:00+02:00
https://konradzemek.com/2013/07/22/gsoc-status-update-week-5
<blockquote>
<p>This is a status update for my <a href="https://www.google-melange.com/gsoc/homepage/google/gsoc2013">Google Summer of Code 2013</a> project -
implementing advanced statistics importers for <a href="https://amarok.kde.org">Amarok</a>. Please read <a href="/2013/06/15/hello-world-me-the-blog-and-gsoc/">the
first post</a> if you would like to know more about the project.</p>
</blockquote>
<p>I'm happy to report that every component of the Importers infrastructure is now
working. There were no major design changes from what I described in my <a href="/2013/07/15/gsoc-status-update-week-4/">last
status update</a>. The most significant minor-ish ones are:</p>
<ul>
<li>provider creation and configuration dialogs management was moved from
<code>StatSyncing::Controller</code> to the GUI (but the <code>Controller</code> is still tasked
with instantiating the dialog),</li>
<li><code>ImporterFactory</code> was renamed to <code>ImporterManager</code> to better reflect its
responsibilities.</li>
</ul>
<p>Otherwise, I implemented what remained to be implemented, squashed some small
bugs and added some polish. What I like to call "Importers framework" (but what
is probably too monumental of a name) is ready for service, although it will be
polished and improved still throughout the summer.</p>
<h2 id="8076-words">8076 words</h2>
<!-- _includes/gallery.html -->
<div class="gallery">
<div>
<figure class="figure-2">
<a href="/assets/2013/07/pluginEnabled.png" target="_blank">
<img src="/assets/2013/07/pluginEnabled.png" alt="One plugin now, three left to go"/>
</a>
<figcaption>One plugin now, three left to go</figcaption>
</figure>
</a>
<figure class="figure-2 figure-margin">
<a href="/assets/2013/07/metadataPluginDisabled.png" target="_blank">
<img src="/assets/2013/07/metadataPluginDisabled.png" alt="The plugin is disabled – related providers are disconnected. Note that the "Add" button is grayed out"/>
</a>
<figcaption>The plugin is disabled – related providers are disconnected. Note that the "Add" button is grayed out</figcaption>
</figure>
</a>
</div>
<div>
<figure class="figure-2">
<a href="/assets/2013/07/metadataPluginEnabledConfigure.png" target="_blank">
<img src="/assets/2013/07/metadataPluginEnabledConfigure.png" alt="The plugin is now enabled. The "Add" button is enabled as well, and we can configure selected provider"/>
</a>
<figcaption>The plugin is now enabled. The "Add" button is enabled as well, and we can configure selected provider</figcaption>
</figure>
</a>
<figure class="figure-2 figure-margin">
<a href="/assets/2013/07/createTargetDialogMySQL.png" target="_blank">
<img src="/assets/2013/07/createTargetDialogMySQL.png" alt="Provider creation dialog – configure the new provider"/>
</a>
<figcaption>Provider creation dialog – configure the new provider</figcaption>
</figure>
</a>
</div>
<div>
<figure class="figure-2">
<a href="/assets/2013/07/configureDialog.png" target="_blank">
<img src="/assets/2013/07/configureDialog.png" alt="The provider configuration dialog"/>
</a>
<figcaption>The provider configuration dialog</figcaption>
</figure>
</a>
<figure class="figure-2 figure-margin">
<a href="/assets/2013/07/createTargetDialog.png" target="_blank">
<img src="/assets/2013/07/createTargetDialog.png" alt="Provider creation dialog – choose provider type"/>
</a>
<figcaption>Provider creation dialog – choose provider type</figcaption>
</figure>
</a>
</div>
<div>
<figure class="figure-2">
<a href="/assets/2013/07/synchronizeDialog.png" target="_blank">
<img src="/assets/2013/07/synchronizeDialog.png" alt="StatSyncing Synchronization dialog, choose providers to synchronize"/>
</a>
<figcaption>StatSyncing Synchronization dialog, choose providers to synchronize</figcaption>
</figure>
</a>
<figure class="figure-2 figure-margin">
<a href="/assets/2013/07/synchronizeDialog2.png" target="_blank">
<img src="/assets/2013/07/synchronizeDialog2.png" alt="StatSyncing Synchronization dialog – it works! Disclaimer: song ratings are test values"/>
</a>
<figcaption>StatSyncing Synchronization dialog – it works! Disclaimer: song ratings are test values</figcaption>
</figure>
</a>
</div>
</div>
<h2 id="the-future">The future</h2>
<p>This week I'm going to start and finish implementing iTunes importer. After that
I'm going to spend some time on tests, repurposing them for the new Importers
framework. If most things go well, everything that I have planned should be
ready for the GSoC Midterm Evaluation, along with some things I haven't (i.e.
the "framework").</p>
<p>After that, well, I don't know yet. I will get a taste of implementing a
concrete importer this week, and will see how much work it requires right now.
Depending on that, more importing targets than just Amarok 2.x and Rhythmbox may
be expected from the second half on my project.</p>
<p>As always, you can check out my progress on my <a href="https://quickgit.kde.org/?p=clones%2Famarok%2Fkzemek%2Famarok.git">public Amarok clone</a>. The branch
is named <code>gsoc-importers</code>.</p>
<p>Thanks for reading!</p>
GSoC Status Update - Week 4
2013-07-15T20:16:00+02:00
https://konradzemek.com/2013/07/15/gsoc-status-update-week-4
<blockquote>
<p>This is a status update for my <a href="https://www.google-melange.com/gsoc/homepage/google/gsoc2013">Google Summer of Code 2013</a> project -
implementing advanced statistics importers for <a href="https://amarok.kde.org">Amarok</a>. Please read <a href="/2013/06/15/hello-world-me-the-blog-and-gsoc/">the
first post</a> if you would like to know more about the project.</p>
</blockquote>
<p>Let me clear some things out first. Scheduled: <strong>"Reimplementing FastForward
importer. At the end of the week I expect to have a functional importer,
possibly with some quirks to iron out. Implementing unit tests to assert
correctness."</strong> Done: I could argue the former; the Amarok 1.4 importer <em>does</em>
work, with some quirks to iron out. The latter? Tests are untouched.</p>
<p>Yet this was by far the most intensive week of coding so far, and the task I
focused on is the hardest part - the <em>most important part</em> - of the whole
project. First, to give some context, let's talk about how I design programming
frameworks.</p>
<h2 id="how-do-i-do-it">How do I do it</h2>
<p>I'm not much of a fan of the <a href="https://en.wikipedia.org/wiki/Waterfall_model">waterfall model</a>. All of its other shortcomings
aside, I need to see how things actually <em>work</em> together, what is the overall
<em>feel</em> of the system. Designing by gut feeling may seem strange, but I'm sure
that many programmers will know what I'm talking about here; sometimes a
particular mechanism just feels awkward, particular responsibility ill-placed,
method redundant. This "methodology" is probably described and named, somewhere.
It's something like "incremental development meets rapid prototyping."</p>
<p>I start with a very rough draft of the system, and implement it using empty
classes, stubs, etc. (I like when things compile.) Then I pick a component -
maybe a single class, maybe a small family - and flesh it out. During this
process, I invariably become aware of multiple problems with the sketch, so I
rework it to address them. Then I pick another component, and the process
repeats. On each iteration I end up with a better design, and it can be vastly
different from the previous one.</p>
<p>The moral here is: as not every component is done yet, the system that I wrote
<em>may</em> undergo major changes, and definitely <em>will</em> undergo minor-ish ones. So,
on to the actual design!</p>
<h2 id="the-actual-design">The actual design</h2>
<p>A picture is worth a thousand words, so here's a simple class diagram (please
note that only some associations are displayed; e.g. in reality <code>Controller</code> has
some kind of association with most of shown classes). I <em>wrote</em> this diagram
using <a href="http://plantuml.com">PlantUML</a> - give it a look!</p>
<!-- _includes/image.html -->
<figure class="image-wrapper">
<a href="/assets/2013/07/diagram.png" title="" target="_blank">
<img src="/assets/2013/07/diagram.png" alt="Simple class diagram"/>
</a>
<figcaption class="image-caption">Simple class diagram</figcaption>
</figure>
<p>First, importers are now plugins. That means that they need to be registered
with the plugin framework (using a semi-convenient macro), and they need to
provide a <a href="http://standards.freedesktop.org/desktop-entry-spec/desktop-entry-spec-latest.html">.desktop file</a> containing some meta-information. On the flip side we
get automatic loading, which can be turned on and off at user's leisure.</p>
<!-- _includes/image.html -->
<figure class="image-wrapper">
<a href="/assets/2013/07/amarok-plugin-display.png" title="Amarok plugin display" target="_blank">
<img src="/assets/2013/07/amarok-plugin-display.png" alt="Hey, it's a plugin! (Please disregard the description)"/>
</a>
<figcaption class="image-caption">Hey, it's a plugin! (Please disregard the description)</figcaption>
</figure>
<p>The plugin infrastructure takes care of instantiating and initializing
<code>Plugins::PluginFactory</code> class, which in case of importers is a subclass of
<code>ImporterFactory</code>, which in turn is a subclass of <code>ProviderFactory</code>. The
<code>ImporterFactory</code> and <code>ImporterProvider</code> classes are serving only as a common
base for importers, to reduce boilerplate code and duplication - all of the
relevant interface methods are specified in <code>ProviderFactory</code>.</p>
<p>The <code>ProviderFactory</code> contains, among others, methods returning: icon, name,
description, and configuration widget for that particular provider type. The
configuration widget is of type <code>ProviderConfigWidget</code>, which is an interface
with one method: <code>config()</code>, returning a <code>QVariantMap</code> populated with values set
by user. The returned <code>QVariantMap</code> is in turn used by another method of
<code>ProviderFactory</code>, <code>createProvider( const QVariantMap &config )</code>.</p>
<!-- _includes/image.html -->
<figure class="image-wrapper">
<a href="/assets/2013/07/sequence.png" title="" target="_blank">
<img src="/assets/2013/07/sequence.png" alt="Provider creation sequence diagram"/>
</a>
<figcaption class="image-caption">Provider creation sequence diagram</figcaption>
</figure>
<p>I've also extended <code>StatSyncing::Provider</code> with two methods:
<code>ProviderConfigWidget *configWidget()</code>, which returns a configuration widget for
this provider instance, if any; and <code>bool isConfigurable()</code>, which returns
<code>true</code> if provider is configurable. The reason for this is to enable
reconfiguration of importers, and putting it in a base class makes sense (in
future it could be used for Last.fm provider, for instance). The implementer of
<code>StatSyncing::Provider</code> is responsible for reconfiguring the provider instance
after the configuration is done. The question of <em>how</em> he will know that it's
done will be answered later, but reconfiguring can be done in much the same way
that initial configuration is done.</p>
<!-- _includes/image.html -->
<figure class="image-wrapper">
<a href="/assets/2013/07/new-buttons-metadata.png" title="" target="_blank">
<img src="/assets/2013/07/new-buttons-metadata.png" alt="New buttons, and an added synchronization target"/>
</a>
<figcaption class="image-caption">New buttons, and an added synchronization target</figcaption>
</figure>
<p>As I mentioned, <code>ImporterProvider</code> and <code>ImporterFactory</code> serve to reduce
boilerplate. They have some sane defaults which greatly reduce amount of code
needed to implement new providers, and I hope to reduce it yet further. For
starters, <code>ImporterProvider</code> by default delegates queries for description, icon,
configuration widget, to corresponding <code>ImporterFactory</code>. That means many
attributes need only be set in the <code>ImporterFactory</code> subclass, and not in the
Provider, and it will just work. While the reconfiguration part is not yet
programmed in, it will be done automatically by the framework; the same
<code>ProviderConfigWidget</code> will be used, prepopulated by <code>ImporterFactory</code>. The
factory will then create new provider using the modified config, and
transparently swap it with the old provider, immutable-style.</p>
<h2 id="current-status">Current status</h2>
<h3 id="what-works">What works</h3>
<ul>
<li>FastForward Provider</li>
<li>registering plugins</li>
<li>provider creation dialog, complete with creating new provider</li>
<li>loading up configuration at startup</li>
</ul>
<h3 id="what-doesn-39-t">What doesn't</h3>
<ul>
<li>saving configuration at shutdown</li>
<li>reconfiguring provider</li>
<li>forgetting provider</li>
<li>dialogs being pretty - there's a reason for the lack of configuration dialogs'
screenshots</li>
<li>FastForward Provider - the tracks are found but displayed as unique, even
though the metadata seems identical</li>
</ul>
<h3 id="plans-for-the-future">Plans for the future</h3>
<p>Obviously my schedule is now a bit skewed, as by now I was supposed to finish up
FastForward Provider complete with tests, and start implementing iTunes
Provider. Right now the plan is to finish up the importers framework, so
everything that's needed is <em>usable</em> - doesn't have to be perfect yet. If things
will go as I anticipate, implementing iTunes Provider should be a breeze, and
creating every subsequent provider even easier.</p>
<p>As always, you can check out my progress on my <a href="https://quickgit.kde.org/?p=clones%2Famarok%2Fkzemek%2Famarok.git">public Amarok clone</a>. The branch
is named <code>gsoc-importers</code>.</p>
<p>Thanks for reading!</p>
GSoC Status Update – Week 3
2013-07-08T15:02:00+02:00
https://konradzemek.com/2013/07/08/gsoc-status-update-week-3
<blockquote>
<p>This is a status update for my <a href="https://www.google-melange.com/gsoc/homepage/google/gsoc2013">Google Summer of Code 2013</a> project -
implementing advanced statistics importers for <a href="https://amarok.kde.org">Amarok</a>. Please read <a href="/2013/06/15/hello-world-me-the-blog-and-gsoc/">the
first post</a> if you would like to know more about the project.</p>
</blockquote>
<p>For the previous week I had scheduled <strong>"Reimplementing FastForward importer,
cleaning up data-reading module. At the end of the week I expect to have working
implementation of FastForwardProvider."</strong> That's done now, and I can't say it
took a lot of coding.</p>
<p>It took a fair amount of thinking, though. Classes that inherit
<code>StatSyncing::Provider</code> and <code>StatSyncing::Track</code> have to fulfill certain
conditions for working in multi-threaded environment. In short, the former must
be reentrant, and the latter thread-safe.</p>
<p>Surprisingly, there are various definitions of reentrancy and thread-safety.
<a href="http://qt-project.org/doc/qt-4.8/threads-reentrancy.html">The Qt's one</a> basically boils down to this: a class is reentrant
when different threads can safely work on different objects, and thread-safe
when they can work on the same object. [Wikipedia disagrees]<a href="https://en.wikipedia.org/wiki/Reentrancy_%28computing%29">Wikipedia
reentrancy</a>. In it's definition, a method is reentrant when it can be
interrupted in the middle and started again. In that sense, the function in the
following example (courtesy of Wikipedia) is thread- safe, but not reentrant; an
invocation interrupted in the function body would leave the mutex locked, and
any subsequent invocation would hang on <code>mutex_lock()</code>.</p>
<div class="highlight"><pre><code class="language-c++" data-lang="c++"><span class="kt">int</span> <span class="nf">function</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">mutex_lock</span><span class="p">();</span>
<span class="c1">// ...</span>
<span class="n">function</span> <span class="n">body</span>
<span class="c1">// ...</span>
<span class="n">mutex_unlock</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div>
<p>But leaving this small tangent aside, that wasn't the only thing I had to think
through. I had a small case of premature optimization going on: I didn't want to
open a new SQL connection for every request (Amarok stores it's main
collection's data in SQL database). It sounds simple enough to do, but coupled
with the fact that <a href="http://qt-project.org/doc/qt-4.8/threads-reentrancy.html">"a connection can only be used from within the thread that
created it"</a>, and multithreaded nature of providers, this was an
interesting problem to get right. So I rediscovered what every other Qt
developer already knew: <strong>if you need something to run in a specific thread, use
Signals & Slots mechanism</strong>. With <code>Qt::BlockingQueuedConnection</code> and using <code>emit
foo()</code> instead of <code>slotFoo()</code>, we basically get an effect of executing a
function - synchronously - in a different thread, and with an added benefit of
not having to bother with synchronization ourselves.</p>
<p>It gets a little bit more awkward when both caller and callee can happen to be
in the same thread, as <code>Qt::BlockingQueuedConnection</code> blocks caller's thread
until called slot is done:</p>
<div class="highlight"><pre><code class="language-c++" data-lang="c++"> <span class="n">connect</span><span class="p">(</span> <span class="k">this</span><span class="p">,</span> <span class="n">SIGNAL</span><span class="p">(</span><span class="n">retrieveAllData</span><span class="p">()),</span> <span class="n">SLOT</span><span class="p">(</span><span class="n">slotRetrieveAllData</span><span class="p">()),</span> <span class="n">Qt</span><span class="o">::</span><span class="n">BlockingQueuedConnection</span> <span class="p">);</span>
<span class="c1">// ...</span>
<span class="k">if</span><span class="p">(</span> <span class="kr">thread</span><span class="p">()</span> <span class="o">==</span> <span class="n">QCoreApplication</span><span class="o">::</span><span class="n">instance</span><span class="p">()</span><span class="o">-></span><span class="kr">thread</span><span class="p">()</span> <span class="p">)</span>
<span class="n">slotRetrieveAllData</span><span class="p">();</span>
<span class="k">else</span>
<span class="n">emit</span> <span class="nf">retrieveAllData</span><span class="p">();</span>
</code></pre></div>
<p>But hey, the pros outweigh the cons, and it's what matters. And after I was done
with <code>FastForwardProvider</code> and <code>FastForwardTrack</code>, I decided to add some lazy
initialization to the mix...</p>
<p>So, to wrap this up. Aside from multi-threaded fun and figuring out the right
file system hierarchy, I also spent a lot of time thinking about the bigger
picture, that is how and when importers would actually load up the configuration
and register themselves with right controllers. It's something I initially
overlooked in my proposal, and something I'll try to address this week, possibly
skewing my schedule a little bit. More on this topic coming next week.</p>
<p>Oh, and I'm not done with <code>FastForwardProvider</code>, not really. It's more of a
prototype. As more providers come to life, they will see a lot of changes aiming
at deduplication and easing the creation of new ones.</p>
<p>Thanks for reading!</p>
GSoC Status Update - Week 2
2013-07-01T23:52:00+02:00
https://konradzemek.com/2013/07/01/gsoc-status-update-week-2
<blockquote>
<p>This is a status update for my <a href="https://www.google-melange.com/gsoc/homepage/google/gsoc2013">Google Summer of Code 2013</a> project -
implementing advanced statistics importers for <a href="https://amarok.kde.org">Amarok</a>. Please read <a href="/2013/06/15/hello-world-me-the-blog-and-gsoc/">the
first post</a> if you would like to know more about the project.</p>
</blockquote>
<p>I didn't think writing tests could be so much fun! Neither did I think they
could take me so much time. What I scheduled for the past week was
<strong>"implementing the initial test suite for importers (exam week),"</strong> and it was
supposed to be a fast & easy task, allowing me to focus on my exams. And yet
after my last test that week I found myself spending every day intensively
coding. Most of the time was spent on figuring out how to approach testing and
writing the code to make that chosen approach <em>possible</em>; writing tests
themselves actually <em>was</em> the fast & easy task I thought it to be.</p>
<p>At the end, I ran <code>git diff --stat</code> for my changes: <code>26 files changed, 1246
insertions(+), 69 deletions(-)</code>. I was surprised.</p>
<p>I managed to test things the way I wanted to. I created two test suites,
<code>TestITunesImporter</code> and <code>TestFastForwardImporter</code>, both inheriting from
<code>TestImportersCommon</code>, which holds common tests and utility functions. I
<em>slightly</em> modified both importers so that I could statically provide them with
arbitrary database paths to read from and arbitrary collections to write to -
one of my goals for this week was *not to modify existing importers in any major
way.</p>
<p>I created two collections: <code>localCollection</code> and <code>fileCollection</code>; the latter
fulfilling a role of <code>FileTrackProvider</code>, giving importers information about
tracks located at nonexistent paths; the former being the collection tracks were
synchronized to. To easily check, clear, and modify their contents I extended
capabilities of <code>Collections::CollectionTestImpl</code> class, which stores collection
in-memory. The 'init' and 'clean' test procedures take care of filling the
<code>fileCollection</code> with necessary data, clearing <code>localCollection</code>, and resetting
statistics after each test.</p>
<p>The code has no idea which importers it's dealing with, operating on
<code>DatabaseImporter</code>, a superclass of both. The tests <em>don't know even that</em>. A
test sets preconditions by modifying collections' contents, calls
<code>blockingImport()</code> method of <code>TestImportersCommon</code>, and then checks the
resulting state. An elegant, implementation-agnostic way, which fulfills my
second goal for the week: <em>write tests for current importers in a way that would
make them easily reused for reimplemented importers</em>. Done and done.</p>
<p>So here's something I learned that week: how to run an asynchronous task in Qt
and block until it's done. That's a useful thing for testing! Here's the full
implementation of <code>blockingImport()</code>:</p>
<div class="highlight"><pre><code class="language-c++" data-lang="c++"><span class="kt">void</span>
<span class="n">TestImportersCommon</span><span class="o">::</span><span class="n">blockingImport</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">QScopedPointer</span><span class="o"><</span><span class="n">DatabaseImporter</span><span class="o">></span> <span class="n">importer</span><span class="p">(</span> <span class="n">newInstance</span><span class="p">()</span> <span class="p">);</span>
<span class="n">QEventLoop</span> <span class="n">loop</span><span class="p">;</span>
<span class="n">connect</span><span class="p">(</span> <span class="n">importer</span><span class="p">.</span><span class="n">data</span><span class="p">(),</span> <span class="n">SIGNAL</span><span class="p">(</span><span class="n">importSucceeded</span><span class="p">()),</span> <span class="o">&</span><span class="n">loop</span><span class="p">,</span> <span class="n">SLOT</span><span class="p">(</span><span class="n">quit</span><span class="p">()),</span> <span class="n">Qt</span><span class="o">::</span><span class="n">QueuedConnection</span> <span class="p">);</span>
<span class="n">connect</span><span class="p">(</span> <span class="n">importer</span><span class="p">.</span><span class="n">data</span><span class="p">(),</span> <span class="n">SIGNAL</span><span class="p">(</span><span class="n">importFailed</span><span class="p">()),</span> <span class="o">&</span><span class="n">loop</span><span class="p">,</span> <span class="n">SLOT</span><span class="p">(</span><span class="n">quit</span><span class="p">()),</span> <span class="n">Qt</span><span class="o">::</span><span class="n">QueuedConnection</span> <span class="p">);</span>
<span class="n">connect</span><span class="p">(</span> <span class="n">importer</span><span class="p">.</span><span class="n">data</span><span class="p">(),</span> <span class="n">SIGNAL</span><span class="p">(</span><span class="n">importError</span><span class="p">(</span><span class="n">QString</span><span class="p">)),</span> <span class="o">&</span><span class="n">loop</span><span class="p">,</span> <span class="n">SLOT</span><span class="p">(</span><span class="n">quit</span><span class="p">()),</span> <span class="n">Qt</span><span class="o">::</span><span class="n">QueuedConnection</span> <span class="p">);</span>
<span class="n">connect</span><span class="p">(</span> <span class="n">importer</span><span class="p">.</span><span class="n">data</span><span class="p">(),</span> <span class="n">SIGNAL</span><span class="p">(</span><span class="n">importFailed</span><span class="p">()),</span> <span class="k">this</span><span class="p">,</span> <span class="n">SLOT</span><span class="p">(</span><span class="n">importFailed</span><span class="p">())</span> <span class="p">);</span>
<span class="n">connect</span><span class="p">(</span> <span class="n">importer</span><span class="p">.</span><span class="n">data</span><span class="p">(),</span> <span class="n">SIGNAL</span><span class="p">(</span><span class="n">importError</span><span class="p">(</span><span class="n">QString</span><span class="p">)),</span> <span class="k">this</span><span class="p">,</span> <span class="n">SLOT</span><span class="p">(</span><span class="n">importFailed</span><span class="p">())</span> <span class="p">);</span>
<span class="n">importer</span><span class="o">-></span><span class="n">startImporting</span><span class="p">();</span>
<span class="n">loop</span><span class="p">.</span><span class="n">exec</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div>
<p>And this is as personal as it gets when it comes to interacting with importers
in tests. (As you can see, I'm playing with a new plugin for code snippets.
Configuring it fully is yet another thing I have planned for later.)</p>
<p>You can check out my progress on my <a href="https://quickgit.kde.org/?p=clones%2Famarok%2Fkzemek%2Famarok.git">public Amarok clone</a>. The branch is named
<code>gsoc-importers</code>.</p>
<p>Thanks for reading!</p>
GSoC Status Update - Week 1
2013-06-24T20:20:00+02:00
https://konradzemek.com/2013/06/24/gsoc-status-update-week-1
<blockquote>
<p>This is a status update for my <a href="https://www.google-melange.com/gsoc/homepage/google/gsoc2013">Google Summer of Code 2013</a> project -
implementing advanced statistics importers for <a href="https://amarok.kde.org">Amarok</a>. Please read <a href="/2013/06/15/hello-world-me-the-blog-and-gsoc/">the
first post</a> if you would like to know more about the project.</p>
</blockquote>
<p>Long days and pleasant nights.</p>
<p>The first week of GSoC has passed. What I had scheduled for that week (and I'm
still getting around to placing the schedule around here) was "Drafting up test
cases for FastForward and iTunes importers."</p>
<p>I admit that it doesn't sound too exciting, and neither does the next one which
is all about implementing these drafted tests. The reason for that is that I'm
currently in the middle of final exams; as of writing this posts, three of them
are behind me and three are left to go. The last exam is July 2nd, so all of the
real fun will start after that.</p>
<p>So, let's write about what I actually did that week, which amounts to this: I
went through the current importers implementation and noted what they currently
do. I then compared this with what the reimplemented importers should do, and
then noted on my whiteboard what kind of thing I should then be testing. I also
familiarized myself with QTestLib and KDE helper methods for testing and
prepared a blank test suite, so I'm basically all set to implement things. And
that's about all.</p>
<p>I want to write a bit about <em>why</em> I want to implement these tests at all. Their
purpose isn't to test the implementation of current importers, because they're
being replaced anyway. What I want to do is assert how the logic works right
now, and then carry on these assertions to the new implementation. The bottom
line is: they'll make sure that the new implementation is no worse than the old
one, which is one of my project's goals. :)</p>
<p>Signing off before I write everything I have about tests, and then have nothing
to write about next week. Thanks for reading!</p>
Hello World! Me, the blog, and GSoC
2013-06-15T22:22:00+02:00
https://konradzemek.com/2013/06/15/hello-world-me-the-blog-and-gsoc
<p>Hey! My name is Konrad Zemek, I'm a student at AGH University of Science and
Technology, Kraków, Poland. I study Computer Science, currently finishing my
second year. I'm a <em>programmer</em>. Mainly a C++ programmer, but that's just
because I write in C++ professionally - I know a few other languages and I'm
fast to pick up new ones. I could write about this a lot more, but I guess it
all just comes to that: I'm a programmer, and a good one.</p>
<p>And I have other traits, too! I love gaming, particularly video gaming. I eat
through books, mostly of the fantasy kind. I ride a bicycle almost every single
day, and I'm learning to play the electric guitar, dreaming that I could someday
justify buying a <a href="http://assets.fender.com/frl/1e81a4473f5b6d3b790bee5b5e61aa7d/generated/86ceb3c861ac3d7b0737b2ccde488ca3.png">Stratocaster</a>.</p>
<p>I'll get to placing my photo, not made hastily with a cellphone, somewhere
around here - in the meantime I redirect you to <a href="https://plus.google.com/113823919880489418232">my Google+ page</a>.</p>
<h2 id="the-blog">The blog</h2>
<p>I really tried not to commit too much time to deploy this blog, as I'm behind my
schedule already. And believe me, it was hard - I'm the type of person who
spends weeks reading about color theory and typography before deploying a
website. This time, at least from the visual side, everything is pretty generic
- still, I see great many hours of tweaking and customizing this blog in my
future.</p>
<p>Just not now.</p>
<p>It's a bit more interesting from the technical side. I deployed the WordPress
application on the Amazon cloud, AWS; its code (the "application" part) resides
in a git repository, which I push to "Elastic Beanstalk" (a Platform-
as-a-Service) after every update, which in turn spins up a generic instance of
GNU/Linux and deploys the application there. Nothing on this instance is
persistent, so no actual <em>content</em> can be stored there. The blog connects to a
SQL database, which is provided by another part of AWS. Then there's persistent
content that is neither a part of the application nor it belongs in the
database, like uploaded images and other post attachments - these are stored by
Amazon S3 service. <a href="https://cloudflare.com">CloudFlare</a> provides DNS, caching, and some anti-bot
screening.</p>
<p>There are a lot of things on the technical side that I'm itching to talk about,
but I've got more important thing to write about, that being...</p>
<h2 id="google-summer-of-code">Google Summer of Code</h2>
<p>The main purpose of this blog, or rather the reason it came into existence, is
to write about <a href="https://www.google-melange.com/gsoc/homepage/google/gsoc2013">Google Summer of Code</a>, more specifically about my own <a href="https://www.google-melange.com/gsoc/project/details/google/gsoc2013/kzemek/5662278724616192">GSoC
project</a>. The title of this proposal is <strong>Reimplement Amarok 1.4 (FastForward) &
iTunes importers on top of Statistics Synchronization framework, and add Amarok
2.x and Rhythmbox as synchronization targets.</strong> <a href="https://amarok.kde.org">Amarok</a> is a legendary music
player, part of the KDE software suite (I'd say it's a Linux music player, but
that's not entirely true).</p>
<p>Every but the most basic music player collects data about music being played -
it's called personal metadata, and includes information like number of times a
given track has been played, or user's rating of the track. For users who use
those features, it's quite a big deal - it allows for playing favorite tracks,
or maybe the tracks which were not listened to enough, all without any need to
spend time setting up custom playlist. Or perhaps you like to share your most-
loved tracks through a service like <a href="https://last.fm">Last.fm</a>? None of that would be achievable
without personal metadata.</p>
<p>Currently, if you're using iTunes or an old Amarok version, you are able to
share this metadata with Amarok, although there's no easy way to keep it
synchronized. At a basic level, my project aims to add that very capability - to
easily resynchronize Amarok with iTunes, previous Amarok versions, and
Rhythmbox, Ubuntu's default music player. There are also some stretch goals,
like being able to do a two-way synchronization (update metadata on the other
player), and I'm also quite confident of reaching those. I will post my weekly
progress updates here during the course of next three months, and I think my
planned schedule will end up in a widget somewhere on this site. There also will
be more details coming. I'll get to it, I promise!</p>
<p>So that's me done for this post, before I run out of things to say in the next
one. Thanks for reading. :)</p>