Twothousandandeighteen

2018 was not bad at all. Here are some of the major achievements and events of last year. Quite a few things happened during the last year.

  • After I bought a 4K TV, I finally got my XBOX One X as there was a great deal at Amazon. Finally, I was able to fully utilize my TV.
  • In February, I started teaching Interactive Systems and MCI at the Baden-Württemberg Cooperative State University in Karlsruhe. It looks like the course went quite well as the University decided to offer it a second time during the autumn where I gave the lecture a second time. As a fun fact, they forgot to tell me until I got a note some days before the course was scheduled.
  • In May I started teaching a second lecture covering data structures and algorithms, also at the Baden-Württemberg Cooperative State University in Karlsruhe which will be given annually.
  • As a result of the lecture, I started to write a book covering this topic using .NET Core. However, I am afraid I won’t be able to finish it in 2019.
  • I also was appointed as a member of the examination board of Baden-Württemberg Cooperative State University in Karlsruhe where I advised two students writing their bachelor thesis.
  • Also in May, I became a certified Product Manager after becoming a Certified Scrum Product Owner in 2017. The course, given by Product Focus based in London, UK is highly recommended.
  • Our son celebrated his very first birthday and started walking this year. Not really my personal achievements but something you are proud of as a dad.
  • In September we switched from HomeKit to Amazon Alexa and changed meanwhile almost all our Lights to Hue and IKEA TRÅDFRI.
  • There is only a little I was able to achieve in my day job, however, our team managed to finish a five-year multi-million migration.
  • As some side projects, I started learning Ansible as well as Docker and moved my 10-year-old handcrafted, homebrewed server to the latest generation of automated container based version.
  • I also started again blogging, especially about my experiences with Docker, Ansible and further Automation.
  • Unfortunately, our rabbits died by the rabbit hemorrhagic disease. Actually, this was a rather sad event. The vaccine is not easily available for this particular and very variant of the virus. It was an event I learned a lot about vaccines, viruses as well as diseases at all.
  • Also, it seems as we experienced one of the longest droughts in Germany. We did not have any rain for weeks or even months.
  • In December, I was appointed at a teaching assignment of Software Engineering of Complex Systems at the University of Heilbronn. I will start this lecture in February ’19 as well as a lab covering software development. I am really looking forward for these new courses.
  • All over the year, I suffered from overall sleep deficiency. I wish I would know where our 16mo takes his energy from.
  • One highlight was definitely visiting my former manager and old friend from Microsoft Research who meanwhile is head of a university.
  • Eventually, I started using Twitter again.

Although I thought 2018 was a rather boring year, it looks as there was a lot going on. Unfortunately, I was not able to read that many books as I am used to, but his might change in 2019. So I am looking forward to an exciting next year and many new things to learn.

Proper Logwatch Configuration using Ansible

On my way setting up a proper monitoring for my server, I just installed Logwatch to receive a daily summary of the what happened recently on the machine. I will have a look into Prometheus, Grafana, Icinga etc. later. However, for now I just wanted a quick summary of the daily “what’s going on on the machine”. Eventually, I had to fix an occur No such file or directory error.

Therefore, I decided to use Logwatch as a lightweight solution to my needs.

Installation Script

The Ansible script to install Logwatch is straight forward:

- name: Install logwatch
apt:
name: logwatch
state: latest
tags:
- logwatch

- name: Create logwatch.conf file for customisations
file:
path: /etc/logwatch/conf/logwatch.conf
state: touch
tags:
- logwatch

- name: E-Mail to
lineinfile:
dest: /etc/logwatch/conf/logwatch.conf
regexp: "^MailTo ="
line: "MailTo = {{ logwatch_email }}"
state: present
tags:
- logwatch

- name: Set detail
lineinfile:
dest: /etc/logwatch/conf/logwatch.conf
regexp: "^Detail ="
line: "Detail = {{ logwatch_detail }}"
state: present
tags:
- logwatch

Configuration & Troubleshooting

I basically set up two parameters, the e-mail as well as the detail level I want for the report. Important to know is the order Logwatch is applying your configuration settings. Following the recommendations, I did not change anything in the configuration file at

/usr/share/logwatch/default.conf/logwatch.conf

rather I decided to copy the file to

/etc/logwatch/conf/

The reason is the order, logwatch is scanning for configuration parameters in the following order. Each step actually overwrites the previous one.

  • /usr/share/logwatch/default.conf/*
  • /etc/logwatch/conf/dist.conf/*
  • /etc/logwatch/conf/*
  • The script command line arguments

Eventually, I ended up in the following error:

/etc/cron.daily/00logwatch:
/var/cache/logwatch No such file or directory at /usr/sbin/logwatch line 634.
run-parts: /etc/cron.daily/00logwatch exited with return code 2

To fix this, avoid copying the original configuration to one of the other places. I did this because I followed some recommendation I received. Instead, I now touch a new configuration file as well as setting the two parameters for MailTo= as well as Detail=. Both are s set using Ansible variables in my scripts. The additional configuration file now looks pretty boring, though:

MailTo = mail@example.org
Detail = Low

You also can provide these parameters when calling the script in the cron job: Using Ansible the modification would look like the following:

lineinfile: 
dest: /etc/cron.daily/00logwatch
regexp: "^/usr/sbin/logwatch"
line: "/usr/sbin/logwatch --output mail --mailto {{ logwatch_email }} --detail {{ logwath_detail }}"
state: present
create: yes

I decided to change the cron job call as one never can be safe from the file changing during package updates. The same should be valid for the configuration file at its origin place.

tl;dr

Setting up Logwatch using Ansible might cause strange “No file or directory”-errors during the cron job call. This can be avoided by applying additional configuration settings at appropriate configuration locations.

Personal DevOps #3

While most of the prerequisites are met for my automated server setup I came across some issues when I started with my very first Ansible playbooks.

First Ansible Playbooks

First of all, I wanted to start with a quite simple ping playbook, to ensure the servers are reachable by Ansible.

# Playbook to ping all hosts 
---
- hosts: all
  gather_facts: false
  tasks:
    - ping:

When I run this script I was immediately confronted with the very first error. I really love when such things happen. Nothing can motivate one more than immediate failures like the following.

FAILED! => {
"changed": false,
"module_stderr": "Shared connection to xxx.xxx.xxx.xxx closed.\r\n",
"module_stdout": "/bin/sh: 1: /usr/bin/python: not found\r\n",
"msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
"rc": 127
}

As I started with a minimal Ubuntu 18.04 LTS installation, there is simply no Python 2 installed. However, to run the Ansible tasks on the node, Python is required. I made use of the raw task in Ansible to update the package lists as well as install the package python-minimal. In addition, I added the package python2.7-apt in this bootstraper as it is needed later on. Once Python has been installed the ping playbook worked without any problems.

# Bootstrap playbook to install python 2 and python-apt
# It checks first so no unecessary apt updates are performed
---
- hosts: all
  gather_facts: False
  
  tasks:
  - name: install python 2
    raw: test -e /usr/bin/python || (apt -y update && apt install -y python-minimal)
  - name: install python-apt 
    raw: test -e  /usr/lib/python2.7/dist-packages/apt || (apt install -y python2.7-apt)

For both packages, I test for the corresponding directories on the node to avoid unnecessary updates.

Note: When testing for a directory on the shell the following line became very handy:

> [ -e /usr/lib/python2.7/dist-packages/apt ] && echo "Found" || echo "Not found"

At a second step, I created a maintenance playbook to update and upgrade the packages on my node.

# Playbook to update Ubuntu packages 
---
- hosts: all
  gather_facts: false
  tasks:
  - name: update and upgrade apt packages
    become: true
    apt:
      upgrade: yes
      update_cache: yes
      cache_valid_time: 86400

Before including the pyhton-apt package to the bootstraper, I got the following error when dry running the playbook.

fatal: [xxx.xxx.xxx.xxx]: FAILED! => {"changed": false, "msg": "python-apt must be installed to use check mode. If run normally this module can auto-install it."}

Conclusion

While this is not any rocket science for sure, I now have a few essential scripts to bring my server to a base level I can start working with.

Personal DevOps #2

Even though, I wanted to do this structured, beginning with this topic is quite a mess. You have to set up everything and pull all pieces together before one can start working properly.

Getting a Server

First of all, I picked up a new root server. I will install Ubuntu 18.04 LTS on this particular one. I found a great offer at Netcup, where I just ordered a new root server with unlimited traffic.

That way, I can start moving bit by bit from my old, handcrafted and home-brewed server to the new one instead of replacing the old server. Once everything works fine, I will migrate the data and change the DNS entries to the new server.

macOS as Ansible Control Machine

When starting with such a project, one should expect problems from the very first moment. For me, it started already when I wanted to install Ansible on macOX. Why I’ve chosen Ansible over Chef and Puppet will be covered later.

There is no native Ansible package for macOS available. Instead, you can use Homebrew to do so. Assuming Homebrew is already installed, the following command should do the job.

> brew install ansible

However, for me, it already ended in some errors:

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools),
missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

As most of the time the latest macOS version Mojave might be the reason, running

> xcode-select --install

the command line tools will be installed as part of Xcode. Once accomplished, the homebrew command should work like a charm. Eventually, Ansible is available on my local laptop which is now my control machine.

Setting up the SSH Access

Usually, I deactivate all password-based logins on a server and allow only RSA-key-based logins. To generate the key, you can follow the instruction on ssh.com.

To upload the key, ssh-copy-id is needed. However, once again this is not available on macOS and you have to install it using

> brew install ssh-copy-id

Now you can upload the key using

> ssh-copy-id -i ~/.ssh/key user@host

So far I have planned this was the only step necessary on the server. All future settings, including deactivating password only access should be set using Ansible scripts.

Setting up a Repository

As a (former) developer, I almost don’t do anything without putting my stuff into a revisioning system. For my Ansible scripts, I decided to set up a private GitHub repository. Although I use GitLab and Subversion at the moment at work as well as running a Subversion server at home, I meanwhile put almost everything int GitHub repositories. Therefore Github comes just in quite handy.

Why Ansible?

For automation, there are several frameworks. One major advantage of Ansible is the fact, only a single so-called control machine is needed. There is no further Ansible infrastructure needed. Once you have installed Ansible as described above you are ready to go. For Puppet, you need again to deal with a server and agents an eventually you stick with running daemons. While this is a feasible approach e.g. to manage my team’s 100 servers at work, this is an overkill for personal use. Same with Chef. That’s the reason, I decided to use Ansible as it seems a quite feasible approach for personal use with little overhead while being an approach with the least attack vectors.

If not being familiar with ansible, I recommend watching the following 14-minute video, which gives you a good overview of Ansible’s capabilities. You might want to skip the last four minutes as it deals with Red Hat’s Ansible Tower which targets enterprises.


Personal DevOps #1

I recently gave a talk, introducing DevOps how to add value by fully automated toolchains. Following this at my daily work, I realized, that my very personal servers are managed overall manually. Every single change is an epic battle. Therefore, I decided to get my hands dirty and to set up a fully automated toolchain for my personal server.

Why!?

I want to learn about the tools, the automation and the processes necessary. Maybe you ask yourself, why one should do this. I simply do this to fetch up with the current technology development. I meet a lot of technical managers who actually have no clue about the technology they are talking about. Personally, I don’t want to be one of those managers.

On the other hand, I have to move to a more recent version of my server’s OS but I don’t want to set up the server again by hand.

About one year ago, I had to re-install the server completely after a configuration mistake, three months ago I had to spend hours and hours to manually clean my WordPress installations due to an infected site.  So there is an actual need as well.

I decided to blog about this adventure, which is especially interesting, as the blog itself has to move at one point to the new server. As usual, I blog to keep notes for myself. However, if you find the articles helpful, feel free to drop me a line, recommendations and tips are warmly welcomed.

Goals

I thought about some goals I want to achieve. Ultimately, I want a fully automated deployment pipeline for at least my main server. To achieve this, I want to

  • upgrade to a new version of my server’s OS
  • apply automation scripts such as Puppet, Chef or Ansible
  • containerize everything – at least Mail and Web-Server have to be deployed independently
  • automatically setup and create my WordPress instances
  • set up a mechanism to easily deploy .NET Core Web APIs in containers
  • create a single place where all “data” lives
  • create a mechanism to backup the data
  • establish monitoring and alerting push changes through the toolchain fully automated
  • run everything through repositories and  a CI/CD pipeline

tl;dr

I am going to replace my server and building a fully automated CI/CD pipeline.

 

Enable Keychain for SSH Connections after macOS Sierra

If you are connecting a lot from via SSH to remote servers, entering passphrases over and over might be a pain. As long as your hardware is secured, you might consider enabling the macOS keychain also for your SSH connections.

Go to ~/.ssh and open or create the config file. Now just enter the following entry to the file.

Host *
    UseKeychain yes

In the case you want to enable the keychain only for a certain server, simply replace * with your hostname such as foo.com.

Prior to macOS Sierra, macOS offered a dialog to enter the passphrase where you were able to select the passphrase to be remembered. For some reason, this dialog was removed. On the other hand, the keychain option has been introduced with macOS 10.12.2.

Preview as PDF Viewer for LaTeX Workshop on Visual Studio Code on macOS

After switching to Visual Studio Code as LaTeX editor, I installed the LaTeX Workshop extension to gain decent support for editing LaTeX files on Code.

After fiddling around with the settings, I still was not able to use Preview on macOS.

First of all, make sure external PDF viewers are enabled in the user settings.

"latex-workshop.view.pdf.viewer": "external"

Furthermore, set Preview es external viewer. The trick here is, don’t think about Preview as a viewer, actually, it is the open command.

Just set up the viewer as you would start preview on a terminal.

open -a preview mybook.pdf
This command used for user settings eventually ends in the following section.
"latex-workshop.view.pdf.external.command": {
    "command": "open",
    "args": [
        "-a",
        "preview",
        "%PDF%"
    ]
}
With these settings, Preview will spawn with the created PDF right out of Visual Studio Code.

WordPress Level Up – Applying Let’s Encrypt to Self-Hosted WordPress

Finally, I found some minutes to set up my website with SSL encryption. The issue here, many hosters demand a fortune for certificates. 

Applying Let’s Encrypt

Let’s Encrypt is a free alternative, providing certificates, accepted by most of the browsers.

While manually installing a certificate can be a real pain, Let’s Encrypt utilizes Certbot to do on your behave. Once installed you can select the sites to protect and let do Certbot its work. There is a crisp description on the Let’s Encrypt page which explains how this actually works.

To be honest, applying SSL certificate using this setup makes it absolutely easy for everybody to do so – as long as you have shell access on your server. After downloading the packages – which are provided for a variety of OS and Web server software – Certbot even takes care of the configuration.  You also can configure the sites in a way, that all HTTP requests are automatically are forwarded to HTTPS.

Once done, the site was already available via HTTPS. Unfortunately, Chrome told me the connection is still not secure.

The help provided did not help much either.

Further investigation eventually showed all images within posts did not use HTTPS even after the base URL of the site was changed in WordPress settings.

The links are not created on the fly – the are actually stored in the text. In the database. At least for internal resources (aka images from your own server), I expected something like relative links or similar. To be honest, I have never looked that much at the WordPress internals.

Altering the Database

To change this quickly, I decided to alter all not secure URLs in the database. As changing the protocol from HTTP to HTTPS is changing base URLs as when changing the domain, you could make use of tools to do so.

I found Misha Rudrastyh’s Query Generator very useful to create all SQL queries for changing the database content.

When moving WordPress websites from one domain to another, this tool is a great time saver. Just generate the queries and run them in MySQL.

In my case, I ended up with the following six statements.

UPDATE wp_options SET option_value = REPLACE(option_value, 'http://www.aheil.de', 'https://www.hack-the-planet.net') WHERE option_name = 'home' OR option_name = 'siteurl';
UPDATE wp_posts SET post_content = REPLACE (post_content, 'http://www.aheil.de', 'https://www.hack-the-planet.net');
UPDATE wp_postmeta SET meta_value = REPLACE (meta_value, 'http://www.aheil.de','https://www.hack-the-planet.net');
UPDATE wp_comments SET comment_content = REPLACE (comment_content, 'http://www.aheil.de', 'https://www.hack-the-planet.net');
UPDATE wp_comments SET comment_author_url = REPLACE (comment_author_url, 'http://www.aheil.de','https://www.hack-the-planet.net');
UPDATE wp_posts SET guid = REPLACE (guid, 'http://www.aheil.de', 'https://www.hack-the-planet.net') WHERE post_type = 'attachment';

Once done the next request already ended up in a valid and secure HTTPS request.

Warning: Do a backup (apply mysqldump) before altering your WordPress database, in case you brick it for whatever reason.

tl;dr

Using certificates issued by Let’s Encrypt you can automatically apply these by using Certbot to secure your website. While doing this I experienced some issues with WordPress as all URLs are stored as plain text in the database. With generated scripts from Misha Rudrastyh’s Query Generator altering the WordPress content to apply HTTPS instead of HTTP is quite easy.

The Open Office Dilemma – Collaboration vs Working Side by Side

During the last 20 years, I worked in quite some companies, studied at universities, was self employed, worked full time remote and travelled a lot to customers. Currently, I do work in an environment where open office is a maxim and fully supported as this environment should encourage collaboration.

The Project Setup

I worked some days from home office but traveled most of the time to the customer. I slept in hotels a lot. When I was in a city where we had an office, usually Friday was “office day” where our team was not at the customer’s site.

We did have shared desks at the office, everybody had an pedestal and took the next free desk.  That was ok. We used the office day a lot to chitchat and to build up relationships with co-workers. We prepared for the week after, installed software, talked to to IT-support, went to lunch. Sometimes during the week we went to the office late so finish something we could not accomplish at the customers site. We stayed there late and had fun. When in the office it almost felt like a hackaton every time. Also managers and the CEOs worked that way. And the hours in the office it felt more like fetching up with your wo-corkers. Sometimes we run out of desks, so we went to a desk together. But this rarely did happen.

On the other side there was the customer’s site. These were just large project areas where a huge amount of workers tried to do things. At that time there were no meeting rooms for most of the projects I was involved. We just did the meetings and hinted down the project goals. As this job was exhausting, I did not care about the office, the project room and the open space. I never thought of “you could be more productive if not interrupted all the time” because he just chased down the project road.

In both cases there was a lot of collaboration. Within the companies offices it was like a family, we helped each other and were happy to see each other. At the customer’s site collaboration was formed due to the urgency of the projects. We where firefighters, stormtroopers and the emergency.  Why did it work that way? The project were ingeniously staffed. If these teams would have used offices, the probably would have moved to the hallway to work together. It was a team thing, though.

Everything worked, because it was a limited situation. Everybody was aware, that the projects will end. The situation will change with the next project. So you do not care that much. Also once you left the project, you are not interested much in its future. Also this sounds harsh, one has to face this.

The Team Room

Once I joined a high performing team in a SMB in the software development field. Their location was a team room, applying scrum , QA, ProductOwners and ScrumMaster did live in the same team room. With twenty people it was a crowded place. While doing pair programming, it was a noisy place as well. The setup was fair, tables were separated side by side. But one did not face the pair in front of the table as they have been separated by partitioning walls. Half of the tales have been empty. For bi-weekly sprints you worked on different places, however everybody had its own desk, drawers and so on. It was a fair environment and you felt home, even if it was noisy all the time.

At one point we had to perform deep thinking tasks that took several days and weeks. Major do overs and architectural changes. At that point some of os moved (still as pairs) to smaller rooms. for a limited time to get these tasks done. This was backed by the team as everybody did know, there tasks will not be done right if one is interrupted all the time. Still, other team members came over or you were sitting from time to time in the team room.

As this was not my preferred environment, it worked. The team room was fine, we delivered an unbelievable amount of output and the team played well together. There were a lot of offsite activities by the team members, and still, some years afterwards we meet every year to see each other.

In terms of collaboration, it was very good, as the team worked as such, but there also were individuals who did “their thing”. We had no phones at all, and did almost not use any e-mail. It was simply not necessary as everybody was colocated.

 The Team Lead in the Team Room Fail

Several years ago, I took part in establishing a new development team for a new product as a team lead. We got an small office with one meeting room, a team room and a separated office for the Product Manager and the CEO. First of all, we kicked the CEO out of this office and away from the neighborhood of the developers. Also my desk, which was placed right in front of the developer desks, was removed. I went to the separated office together with the PM. We had to talk a lot and spent a reasonable time  on the phone. I did a lot of PoCs which needed some deep thinking time. Sometimes I did this work from my home office.

The team on the other hand worked together similar as in the situation above. However, it was much more “civilized”. The team was only a fith of the size and everybody was a specialist in his field. Everybody got a huge amount of time for their work without interruptions. This team performed in an outstanding way and produced an awesome output within few months in a exorbitant rate of quality – after I removed my desk out of the team room.

The Virtual Team

When I joined Microsoft I think in 2004, I joined a overall virtual team. Quarterly meetings in a regional office with our managers and from time to time some events. How could one collaborate this way? To make it clear from the very beginning: This was one of the most freaking awesome teams I have ever been with. E-Mails was on a minimum but chatting was always present. I think my chat log where about a Gigabyte. We hacked technology like insane – – always linked to each other. Coordination with management was on a minimum. We had our goals and did everything to reach and regularly to beat them.

Even though we worked everybody for him- or herself, we collaborated in an outstanding manner. The most impressive memory to this team are our come togethers. They were half party half hackathon, there was no or little welcome and no goodby ceremony. We just met, worked and left with a simple, see you tomorrow – knowing we already talk in the chats half an hour from that point on.

At that point we had this idea of mobile blogging restricted to the size of a SMS. We pitched this idea to an Redmond based product manager who told us, this idea has no potential. A few years later, Twitter emerged. Also I do not remember the PM, I really do hope he regrets this moment, every day since then.

Collaboration in this team was on an unbelievable level. There was a goal for the entire team, not for a single member of it. Management played an overall role in this and still, one of our managers is a good friend of mine today.

The Remote Worker

In 2006, I joined Microsoft Research. At the very beginning as software developer as remote worker.  Over time the team grew. We had members from Russia, Spain, Italy, France, Germany, UK and the US. You know what? Collaboration was on a almost all the time high. We had simple rules for the code, few guidelines, a lot of automation and awesome team members. From time to time we met at conferences or events. On a irregular timebase we met in person in Cambridge, UK as flights were cheap to this location.

We used e-mail, chats and video phone calls. At this time Skype was not used, it was even banned, within Microsoft, so we fought to use it. We used almost every technology available. I was on a kind of 24/7 standby as you knew knew if you get a call from any country in the world. In fact I was once called in the middle of the night with the request to remote setup some demos as they were supposed to presented in front of Bill Gates. I did not kew about this until I got the call. So we fixed it. Another Sunday, I got a call from Redmond. It was my architect, telling me that he is preparing a demo for the next day and I broke the build. So I fixed it. Even I never switched off my phone, this was never really stressful. There were goals we worked on (and usually reached them) and there was a high level of collaboration. Again, the team was almost fully virtual.

The Office Day Fail

Right now, I am in product management role, doing a lot of in-house consultancy regarding enterprise architecture. My employer has an awesome policy of all work must be able to be performed from any place. And the same way we have shared desks and a clean desk policy. So far so good, but than again, there is this 20% home office rate as we do not have desks for all employees at the same time. We have open space offices or 8 open floors. Table groups of four everybody facing each other. The open space is supposed to encourage collaboration. However, everyday I see how many of my co-workers just work side-by-side rather than collaborating. E-Mails and Chats are on a all time high, Mails are written and read during meetings. There are many meetings. The places are crowded, it’s noisy and there are endless interruptions.

There is only little deep thinking you can perform. You are just interrupted that often. The most I was able to sit down 360 seconds before being interrupted and being asked “Do you have five minutes?”. I usually have 20 or more of them after such an interruption.

As there is no barrier as an office, the obstacle to interrupt someone is very low. This in fact does include myself as well. I do interrupt people way to often, which I do regret immediately afterwards.

I love being on the phone or doing Skype calls. I hate it in the office. There are thirty co-workers sitting in your neck. Also I do perform really bad in negotiations in an open space. Behind a door – I do really good I think. I just do not want to bother my co-workers to much. It happened to me that I went to a  toilet to do a Skype call for a negotiation just for having some privacy – and to say things only to be said… behind closed doors.

I don’t know how much I perform in open space in terms of coding and architectural decisions. Creativity drops down to a minimum when being afraid of being interrupted any second. On the other hand I come to the office every morning with new ideas. After sitting just for one or two hours in the evening without being interrupted.

My mails suck. Even though I try to avoid writing e-mails, I happens more than once to me that in the middle of the progress of writing, I was interrupted. Once the interruption was over, and the next interruption just queued up, I accidentally sent the mail. With an unfinished sentence in the middle of the unfinished mail.

When I try to write some code (mostly PoCs or demos nowadays) it takes ages. I either do it before the majority of co-workers arrives or later, when most of them have left.

I take my mandatory home office day. I usually get more intellectual work done during this day than in the rest for the week.

Without exact measurements, I feel like in a disruptive environment, I only perform 25% of what I could achieve otherwise.

If I look at the team, everybody works side by side. Almost no collaboration. Excessive usage of communication tool for remote communication and ticketing systems are utilized rather than collaborating.

Is it Only Me?

I wondered if this only for me. No one seems to have an issue with this. Everybody seems to be happy, beside me. Until a few weeks ago.

Two co-workers told me, that If I want to get something done in code, I should join there every-evening.-coding session. So they set up a Skype conference for one or two hours almost every evening where they actually do the new things.

Another co-worker came by with an awesome presentation. He did an overall comparison of platforms. He showed it to us, and showed us a book he worked through. He did during night hours for the last few weeks.

Actually, these are my top developers. I have to wonder why they have to do this during night hours. When they are not disrupted.

And then I read something here:

If your work environment fosters distractions (commonly known as “open space”), a grand majority your engineers will be stressed to the bone, and probably doing 10% of what they can actually accomplish.

tl;dr Conclusion

Also this is my very personal opinion, I had the best collaboration experience in remote teams – workin remote. Not based on working hours. Working on-site was never that awesome.

Good people will probably always sacrify their free time to fetch up disruption at their work place.

But why does remote work rock that much? In my opinion, you can work when you can perform the best. Who says I work the best at 8 am in the office? Maybe “my creative time” is 11 pm to 2 am.

If you set up an remote working environment,  this might not work out for everybody. There are always individuals not providing sufficient discipline for self-responsible work. However, in my experience, if you set up a remote working team, you usually have only team members in it, capable of providing this kind of work.

This might not work for a given team, but should be considered when creating new ones as already Martin Fouler pointed out in his article.

The fact that you can get a better team by supporting a remote working pattern has become increasingly important during my time in the software business and I expect its importance to keep growing.

Based on my experience, remote first teams can outperform co-located teams due to many aspects.

How to deal with GMAIL Undelivered Mail Returned to Sender

If you run your own mail server, you might end up quite frustrated because the Google relays do reject your mails.

Maybe your mails get rejected by the Google servers with the following message:

Our system has detected that this  message does not meet IPv6 sending guidelines regarding PTR records and authentication. Please review  https://support.google.com/mail/?p=IPv6AuthError for more information.

If you start reading your frustration level might even increase, due to the difficulty to deal with many of the requirements to get your mails to a Google server delivered. If you start digging into the various topics, you might end up even more frustrated as you have more questions as before.

At the very end, you just have to set up a few things to make your mails fully compliant, so even Google’s server do accept them. In addition you will get high quality mails – at least regarding the tech, content is entirely up to you.

I do run a setup of mail, Dovecot, Postfix and SpamAssasin on a Ubuntu server. Therefore, this article will cover the topic on these examples but should be able to be applied to almost any other system – as long as you replace tech X by tech Y.

Testing you Mail

The first issue is how to learn about your mails issues at all. To evaluate your mails, http://www.mail-tester.com turned out to be a perfect platform for me. As you only can evaluate three mails a day for free, you might throw in a few bucks if you in a hurry, or it might simply take some days to you to finish the tests.

How does it work? You get an unique mail address where you send your mail to. Once the mail was received any flaws in the mail will be displayed and explained (which definitely is a huge added value).

10/10 Score at mail-test.com

Believe me or not, I definitely haven’t had a score of 10/10 before.

SPF Record

First make sure, your DNS provides an SPF record. With no or little knowledge, I tried to create one by myself based on Google’s help page about SPF records. Did not work well at all.

I finally ended up with the following SPF entry for my DNS:

v=spf1 mx ip4:92.51.166.153 -all
  • a is just to indicate that there might be scripts sending mails
  • ip4:92.51.166.153 depicts the mails server we are talking about in the entry
  • -all means there only this mail server while the mx indicates that only domain’s MXs are allowed to send mail for the domain while all others are prohibited.

It took me a while to figure out the right (or at least something working) based in the SFP record syntax. So why do we do this? The idea behind SPF is simple, once you have published your policy, the receiving server can check if the mail is compliant with your policy. In my case, mails sent from scripts might be fine, as long as they come from my server. If someone sends a mail from server 154.354.32.2 this mail will be probably not sent by me, as it is not compliant with my policy. Google’s server do these checks, and that’s on of the reasons your mails are rejected by them.

Finally the SPF entries doe look like the following at my DNS:

DKIM – DomainKeys Identified Mail

Next, I started with DKIM. Again, if you start reading at dkim.org, you probably end up reading several IETF RFCs and a lot of question. Reading RFCs is always good and highly recommended by me, but actually did not solve my issue.

The good thing about Ubuntu is, there is an package called OpenDKIM which can be installed via

sudo apt-get install opendkim opendkim-tools

I got relatively lost, when i came to configure OpenDKIM and Postfix until I found the Postfix/DKIM site at the Ubuntu documentation.

First of all you need to add the domain to the /etc/opendkim.conf. You might want to exchange my domain aheil.de to your corresponding one.

Domain    aheil.de
KeyFile   /etc/postfix/dkim.key 
Selector   dkim

Now, I had to generate the key file.

opendkim-genkey -t -s dkim -d aheil.de

If you did it the same way as me, the file is probably located in the wrong directory. So move the file to the location provided above.

mv dkim.private /etc/postfix/dkim.key

Now I went to /etc/default/opendkim. Unlike as described in the Ubuntu documentation, I had to add the listening socket – which seems to be a standard one at Ubuntu.

SOCKET="inet:8891@localhost"

I then went back to /etc/opendkim.conf and added the same port right below the Selector entry.

Domain    aheil.de 
KeyFile   /etc/postfix/dkim.key 
Selector  dkim
Socket    inet:8891@localhost

Now, I went to the /etc/postfix/main.cf file and added the following entries:

milter_default_action = accept
milter_protocol = 2
smtpd_milters = inet:localhost:8891
non_smtpd_milters = inet:localhost:8891

For some reason, I now ended up with two DKIM entries in my outgoing mails. I had to add no_milters to the receive_override_options line in my /etc/postfix/master.cf. I had to fiddle a little bit with the master.cf until everything worked again.

-o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters

Wherever you have created the keyfile, there will be an additional dkim.txt file. It will contain something like

dkim._domainkey IN TXT ( "v=DKIM1; k=rsa; t=y; "
 "p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCZFYWrNHupXvZHvWJAo6aGB6sXYf+kpUIJv+BcP6NFiF9GBy26oYZKt6/ngCXmNAIY9+yReY8UxG5GIm/QnInbXWxwDXyD0MHD8HrhHyVa6JVqTncexm29M5Bbp/u0JI4ToOQwIQqpgTr0t9ADP8i76pbelYQ5IQDOwJRJQ2N1iQIDAQAB" ) ; ----- DKIM key dkim for aheil.de

Creating the corresponding DNS entry out of this seemed to be the most challenging part. You need to create an TXT entry similr to dkim._domainkey.aheil.de, of course using your domain. and you simply need the follwowing text in there

v=DKIM1;
p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCZFYWrNHupXvZHvWJAo6aGB6sXYf+kpUIJv+BcP6NFiF9GBy26oYZKt6/ngCXmNAIY9+yReY8UxG5GIm/QnInbXWxwDXyD0MHD8HrhHyVa6JVqTncexm29M5Bbp/u0JI4ToOQwIQqpgTr0t9ADP8i76pbelYQ5IQDOwJRJQ2N1iQIDAQAB

No quotes, no brackets, no nothing beside the above. This took me quite a while especially as DNS replication is nothing that works in real time.

For me the entry does now look like the following.

Once this is done, start OpenDKIM and restart Postfix so the changes will apply.

sudo service opendkim start
sudo service postfix restat

So far what does this actually mean? When a mail server receives your mail, it now will be signed with a DKIM key. If you check your mail headers, the will look similar to the following one.

By checking the dkim._domainkey DNS entry created before, the mail receiver now can check if the mail signature is authentic. Any mail server not in possession of the private key will probably not be able to sign the mail though, and could be thought of a spam server.

DMARC – Domain-based Message Authentication, Reporting & Conformance

Finally, it is necessary to create a DMARC record. Once SPF and DKIM are setup as above, you proceed similar as with the DKIM DNS entry.

Again you create a TXT revord for your DNS. This time the record is named _dmarc.domain.TDL, e.g. _dmarc.aheil.de in my case.

I created a relatively simple record:

v=DMARC1; p=none

I.e. p=none: the I do not requests any specific action be taken on mail that fails DMARC authentication and alignment.

At the very end, it looks like this for my DNS:

Summary

To get Google accepting mails from your own mail server, only a few steps are necessary. Setting up SPF, DCIM and DKIM are basically all which might be needed if Google rejects mails from your handcrafted mail server. You might want polish your SPF and DMARC records to align your policies with your very personal goals. If you are interested further in e-mail architecture, the book from Kevin Thomas called Email Architecture, Design, and Implementations might be something for your.