If you are not familiar with Apache Kafka or want to learn about it, check out their site!
To give you a rough idea, these are the three core features of Kafka:

Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system.
Store streams of records in a fault-tolerant durable way.
Process streams of records as they occur.
- https://kafka.apache.org/intro

In order to follow the example, you need to have the Kafka command line tools installed.
In brew you can do that by simply running brew install kafka.
If you don’t want to install via brew or are using linux, you can get kafka like this:

1
2
3

curl -OL http://ftp.fau.de/apache/kafka/2.1.0/kafka_2.11-2.1.0.tgz
tar -xzf kafka_2.11-2.1.0.tgz
cd kafka_2.11-2.1.0/

You find the cli commands in the bin/ subdirectory.
All commands used assume that you are using the new-consumer api and that you added the kafka commands to your PATH or running them from the bin/ directory from above.

How to consume messages from a given Kafka Topic?

Start to consume from the latest offset

1

kafka-console-consumer --bootstrap-server <one-of-your-brokers> --topic <topic-to-consume>

Consume from the beginning

1
2
3

kafka-console-consumer --bootstrap-server <one-of-your-brokers> \
                       --topic <topic-to-consume> \
                       --from-beginning

Consume a fixed amount of messages

1
2
3

kafka-console-consumer --bootstrap-server <one-of-your-brokers> \
                       --topic <topic-to-consume> \
                       --max-messages <number-of-messages>

Consume and print the message keys

1
2
3
4

kafka-console-consumer --bootstrap-server <one-of-your-brokers> \
                       --topic <topic-to-consume> \
                       --property print.key=true \
                       --property key.separator="---"

This will print your messages in a format like <message-key>---<message-payload>.

Consume messages encoded in Avro

EDIT: The kafka-avro-console-consumer is not part of the package I linked above. It is part of the confluent suite. You can get it here.
Another cool feature of Kafka is that it plays well with the Apache Avro format. To consume messages encoded in Avro simply run the following command to get the decoded messages.

1
2
3

kafka-avro-console-consumer --bootstrap-server <one-of-your-brokers> \
                            --topic <topic-to-consume> \
                            --property schema.registry.url=http://<your-schema-registry>

Move the offset of a specific consumer group forward by consuming messages

1
2
3
4

kafka-console-consumer --bootstrap-server <on-of-your-brokers> \
                       --topic <topic-to-consume> \
                       --max-messages <number-of-messages/moving the offset forward> \
                       --consumer-property group.id=<consumer-group-to-forward-the-offset> > /dev/null

I hope you can use this to speed up your work. If there are commands that you are frequently using that are not included here, let me know in the comments so I can add them.

I my opinion he does an outstanding job in explaining the meaning of side effects and functional programming in general.
So instead of writing a long blog post, I leave you with this talk. Money quotes:

Hidden inputs and outputs are called ‘side effects’

Functional Programming is eliminating & controlling side effects

– Kris Jenkins

Let me know if things in this talk need clarification!

A higher order function takes a function as an argument or returns a function as its result, or both.

So lets say you work in a big company and you have to calculate the bonuses for the employees. Of course, not everyone gets the same bonus, but you’re too lazy to pass the specific multiplier at every calculation. So you create seperate functions.
In this example the C-level department members get a bonus of 1.5 of their annual salary. While the salaries may differ, the multiplier stays the same.

1
2
3
4
5
6
7
8
9

defmodule Salary do
  def bonus_calculation(percentage) do
    fn(salary) -> salary * percentage end
  end
end

# c_level_bonuses = Salary.bonus_calculation(1.5)
# c_level_bonuses.(100_000)
# => 150000.0

You may be wondering, why our c_level_bonuses function still knows about the percentage since we haven’t explicitly passed that value.
That’s because the function has knowledge about the environment it was created in - it’s a closure.

If you want to see an example for a function that takes another function as an argument, head to my previous post and take a look at the map function.

And in case you feel adventurous, try to find a use case to write a function that takes and returns a function =)

Recursion

When talking about tail call optimization it might be helpful to have a look at what happens when you call a function recursively.
Lets say we have the following function.

1
2
3
4
5
6

defmodule Recur do
  def multiply_forever(a) do
    b = 5
    b * multiply_forever(a)
  end
end

I you feel adventurous run this function and see what happens with your RAM usage(spoiler: it will go up fast).

And that’s exactly what you expect to happen - a function calls itself forever, every time it uses some memory space for the stack to save what values a and b hold on each run.

Tail Call Optimization

Now we are using the same example, with a small modification.

1
2
3
4
5
6

defmodule Recur do
  def multiply_forever(a) do
    b = 5
    multiply_forever(a * b)
  end
end

Your RAM usage will not grow indefinitely.
Why?
Because the last thing we called in our function is just the function itself.

A function is a subject to tail call optimization if the last thing it does is calling itself - nothing but itself.

How does it work?

In the first case the compiler may not know if he needs to keep those a and b values, so they have to be saved before every new function call.
This might be written down like this:

1
2
3
4
5
6
7
8
9
10

call multiply_forever(a)
save a,b
call multiply_forever(a)
    save a,b
    call multiply_forever(a)
        save a,b
        call multiply_forever(a)
            save a,b
            call multiply_forever(a)
            ...

In contrary, in the second example, the compiler knows, that he only has to care about the new argument to multiply_forever.
So he just updates the argument and JUMPS back to when multiply_forever first got called.
That’s the difference - JUMP vs. CALL

1
2
3
4
5
6

multiply_forever(a)
jump back to multiply_forever, update what we know about a
jump back to multiply_forever, update what we know about a
jump back to multiply_forever, update what we know about a
jump back to multiply_forever, update what we know about a
...

That’s basically what you have to know about tail call optimization(imho).
The next section is just for the sake of a more interesting example, feel free to skip it, I hope this article was useful to you.

A “real world example“ - writing your own map function

Lets say you want to write your own Enum module, it will need a map function. So we are going to write it.

First approach that is not suiting tail call optimization:

1
2
3
4
5
6

defmodule MyEnum do
  def map([], fun) when is_function(fun), do: []
  def map([head|tail], fun) do
    [fun.(head) | map(tail, fun)]
  end
end

Now the “optimized” version(disclaimer: there are better ways to implement this for sure):

1
2
3
4
5
6
7
8
9
10

defmodule MyEnum do
  def map(collection, fun) when is function(fun) and is_list(collection) do
    _map(collection, [], fun)
  end

  defp _map([], acc, _fun), do: Enum.reverse(acc) # yeah it's cheating to use Enum here
  defp _map([head|tail], acc, fun) do
    _map(tail, [fun.(head) | acc], fun)
  end
end

As you can see, the tail call version takes more code, is not as straight forward as the first approach and we even have to reverse our result.
This is just to show, that even though tail optimization is necessary and you have to use it, sometimes it takes a litte bit more work.

Merry Christmas

Yeah, uhm, I think I still owe you a post for some dropbox’ish thing to sync files over multiple devices.
At first I was willing to give seafile a try, but after reading this it became a nono for me.

Fortunately there is Syncthing - a peer to peer based syncing solution that runs on almost any platform.
Another plus: there is an android app and a third party iOS app seems to be in development, awesome!

There is already an impressive number of tutorials for installation and setup out there and the syncthing documentation is great. Therefore I’ll only cover the very specific things to my setup.

Setting up the “Server”

Syncthing is no client-server application.
Files will only sync if at least two devices are running and online. To make syncthing a Dropbox alternative I have one device that is online all the time. It always knows the most recent state of my files.
We will use a Raspberry PI 2 running Arch Linux as our always online device.
Consider the following commands to be executed on the PI.

Step 1: Get the Syncthing Package

# Version might have changed when you are reading this.
curl -OL https://github.com/syncthing/syncthing/releases/download/v0.11.21/syncthing-linux-arm-v0.11.21.tar.gz

Step 2: Unpack the Package

mkdir syncthing
tar xfvz syncthing-linux-arm-v0.11.21.tar.gz -C syncthing --strip-components=1

Nice, Syncthing is now unpacked in a folder called syncthing.

Step 3: Make the Web Interface accessible in your local network

Syncthing brings a nice web GUI that lets you configure your peer. By default it is only accessible from the peer running syncthing.
Since I don’t have desktop environment installedon the PI, I want to access the web GUI from my laptop.
In order to get this working we have to tweak the syncthing config a bit.

First, run syncthing once so it creates all the default folders.

cd syncthing/
./syncthing
# hit CTRL + C after everything booted up.

Now edit the syncthing config with your favorite editor.

vim ~/.config/syncthing/config.xml

Change this line

<gui enabled="true" tls="false">
    <address>127.0.0.1:8384</address>
</gui>

to this

<gui enabled="true" tls="false">
    <address>0.0.0.0:8384</address>
</gui>

We can now access the web GUI of the PI via http://IP_ADDRESS_OF_YOUR_PI:8384

Step 4: Run Syncthing on Startup

You may not want to start Syncthing by hand everytime you restart your Raspberry, the guys from Syncthing got your back.

# do the next stuff as root
su
cd syncthing # if your aren't already there
cp syncthing /usr/bin/
chmod a+x /usr/bin/syncthing

cp etc/linux-systemd/user/syncthing.service /usr/lib/systemd/user/
systemctl --user enable syncthing.service

reboot

Syncthing now starts at every boot!

I had some troubles figuring out how to do this proerly so I thought it would be worth mentioning.

Other Devices

If you are running Arch Linux on your machine it is as easy as this

pacman -S syncthing
systemctl --user enable syncthing.service
systemctl --user start syncthing.service

Set up your synced Folders and such

Please see the documentation for all the configuration and Web GUI stuff, it’s very comprehensive.

Conclusion

I am really satisfied with this solution as my Dropbox alternative, everything seems to work well and I hope my
few notes on setting it up on the Raspberry PI will save you some time while setting up your own Syncthing!

Once installed you are ready to go.
Assuming we have the following file structure and want all JPEG files to be resized.

1
2
3
4
5
6

.
├── 1.jpg
├── 2.jpg
├── 3.jpg
├── 4.jpg
└── output

To make them, let’s say 25% smaller, you can run the following command:

1

mogrify -resize 25% *.jpg

You could also run something like:

1

mogrify -resize 1600x1200 *.jpg

Note that mogrify will replace your original files with the resized ones, so consider backing up your originals before running this.

After resizing, now create the gif using

1

convert -delay 35 -loop 0 *.jpg output/animated.gif

and your done, your gif will be placed in the output folder.

Choosing the image

There are many use cases for the Pi and therefore you will find a lot of images, that will try to match your needs. Since I just want to run some server applications on the Pi, that need no GUI at all, I will go for the Arch Linux image. There are surely enough other images that will fit the needs as well or even better, feel free to try them out, the installation steps should be the same.
You can find the Arch and many other images on the download site at raspberrypi.org

Download the image and prepare the SD card

Depending on your operating system, there are different ways to set up the SD card, so it will boot your Linux distribution of choice.
I will try to cover both Linux and OSX by sticking to the command line, since they have many commands in common.(differences will be marked)

First download and extract the image:

1
2
3
4
5

curl -o arch_arm.img.zip -L http://downloads.raspberrypi.org/arch_latest
# if your system is missing curl, install it via your systems package manager
# some examples
# on ubuntu: apt-get install curl
# on mac with brew: brew install curl

After downloading is finished, check the integrity of the image. This can save you a lot of pain!
A corrupted download will make you to have to repeat the whole setup again.
Generate the checksum of your download and compare it to the checksum on the downloads page.

1
2
3

# generating the checksum of the downloaded file
shasum arch_arm.img.zip
# the command could be named sha1sum for some folks

If the checksums are matching, lets extract the zip file.

1

unzip arch_arm.img.zip

Note that the extracted file will be named something like ArchLinuxArm-some-release-date-rpi.img.

Put in the SD card format it with your tool of choice to FAT32 and remove it.

Then do

1

df -h

Now insert the SD card.
Again run

1

df -h

The additional device listed is your SD card.
In my case it’s /dev/disk2, this will differ from machine to machine. So if I use /dev/disk2 in some following commmands just see it as a placeholder for your device name.

The next step will differ a little whether you are using an OSX or a Linux OS.
The SD card needs to be unmounted now.

1
2

# only for mac users
sudo diskutil unmountDisk /dev/disk2

On Linux it should look something like this:

1
2

# only do this when you are on a linux machine
umount /dev/disk2

Flashing the image

Alright, now it’s time to flash the image to the disk.
This step will again slightly differ from OSX to Linux.

1
2
3

# on a mac machine the m for bs=1m is written in lower case.
# using rdisk2 over disk2 will give you a nice speed boost in many cases but it's mac only
sudo dd bs=1m if=ArchLinuxARM-2014.01-rpi.img of=/dev/rdisk2

1
2

# on linux machines the M is written in upper case an the bs is set to 4
dd bs=4M if=/path/to/your/image.img of=/dev/disk2

Ok, since the image is not that big, flashing will be quite fast.
As the final step eject the SD card.

1
2

# mac
sudo diskutil eject /dev/rdisk2

On Linux, clear the write cache, to safely remove the card.

1
2

# linux
sudo sync

Now the moment of truth has come. Pull your SD card out of your computer an put it in your Pi!
Notice: put the SD card in before plugging in the power supply.

Login will be root, root. Change this immediately:

1
2

# after logging in to the Pi run
passwd

Now the Pi should be up and running, awesome!

So I started with buying a Raspberry Pi(Model B) and throwing Owncloud on it.
While installation and setup worked quite fluently, the performance was a desaster. A simple request to the webinterface of Owncloud on my Pi took about 20 seconds, which brings me to the conclusion, that Owncloud will be no fit for the Pi and it’s capabilities.

Fortunately there is a alternative to Owncloud which looks promising plus it’s open source and free for personal use.
It has a free app for Android as well, therefore I will give it a shot.

My next blog posts will be about setting up useful server applications on the Pi such as

• Seafile - private Owncloud-like file sync
• Mumble - encrypted, low-latency voice chat
• maybe Jabber - XMPP

I will also cover and start with:

• Installing openwrt on my WD N600 Router (to setup DDNS with no-ip.org)
• Installing arch linux on the Pi

Disclaimer: I have no clue if the PI can handle the stuff I am going to install =)

I researched some dynamic DNS providers and was surprised to find, that only a few free solutions were left.
In the past I was very happy with dnydns.org, but they stopped their free plans.
Fortunately there is no-ip.org, which is keeping up with free plans for single users.

Sadly, the stock firmware which ships with my N600 only supports tzo and dyndns.org as providers for dynDNS which are not quite the same company I guess.
So no satisfying solution here.
Which forces me install some cool stuff on my router: OpenWrt

According to their website, installation should be straight forward:

1
2
3
4
5
6
7

1) Download the file openwrt-ar71xx-generic-mynet-n600-squashfs-sysupgrade.bin
2) Configure your computers IP address to 192.168.1.10 and connect to a LAN port in the router.
3) Turn the router off.
4) Using a paperclip, press and hold the reset button on the bottom of the router and turn it on. Hold the reset button for at least 15 seconds.
5) On your computer, visit http://192.168.1.1 NOTE: You will not be able to ping this address.
6) Upload the file openwrt-ar71xx-generic-mynet-n600-squashfs-sysupgrade.bin as downloaded earlier.
7) The router will now flash OpenWRT. This will take a couple of minutes to achieve. You can ping 192.168.1.1 and watch for ping replies to see when your router has rebooted into OpenWRT

Alright step 1 is easy. Step 2 can be tricky if you are using some linux-ish OS.
I did it on Mint with editing /etc/network/interfaces and adding following lines:

1
2
3
4
5

# /etc/network/interfaces
iface eth0 inet static
  address 192.168.1.10
  netmask 255.255.255.0
  gateway 192.168.1.1

Did a

1

sudo /etc/init.d/networking restart

after saving the file.

This enables me to connect my machine and the router directly via LAN.
Step 3 to 7 worked flawlessly.
After flashing OpenWrt ist done, you can telnet on the router.

1

telnet 192.168.1.1

Not lets install a web interface called LuCI.
Fortunately there are great step-by-step instructions already there!

1
2
3
4
5
6
7

opkg update
opkg install luci
opkg install luci-ssl
# Start the web server (uHTTPd)
/etc/init.d/uhttpd start
# Enable the web server (uHTTPd)
/etc/init.d/uhttpd enable

It just works, great.

Now we need DDNS support, since it is not avaiable out-of-the-box by luci.

1
2

opkg install luci-app-ddns
opkg install ddns-scripts

The DDNS service was disabled by default, so I turned it on via System -> Startup

Now after refreshing the page there should be a new tab called “Services” at the web interface’s navigation.
You can choose a ton of providers here. As mentioned, I will go ahead with no-ip.org.

After setting up the PI we will need these prerequisites to use our applications from outside our local network, e.g. with the smartphone.

But this should be it for now.

Please do not forget to set up a root password and all the other stuff like ssh and the wifi(if needed).

1

:sh

in normal mode. If you type exit or hit CTR+D in your shell, you will get back to Vim. I like this behaviour when I am on a remote server via ssh.

Therefor I binded CTR+D in Vim to execute :sh, so I can toggle between shell and Vim with this shortcut.
To get it, include this snippet in your vimrc.

1
2

# ~/.vimrc
noremap <C-d> :sh<cr>