Twelve Factors App

devops

What principles to follow when building modern applications? Twelve Factors Apps is a collection of best-practices.

In the modern era, software is commonly delivered as a service: called web apps, or software-as-a-service. The twelve-factor app is a methodology for building software-as-a-service apps that.

In a nutshell, it covers the following aspects:

I. Codebase
One codebase tracked in revision control, many deploys

II. Dependencies
Explicitly declare and isolate dependencies

III. Config
Store config in the environment

IV. Backing services
Treat backing services as attached resources

V. Build, release, run
Strictly separate build and run stages

VI. Processes
Execute the app as one or more stateless processes

VII. Port binding
Export services via port binding

VIII. Concurrency
Scale out via the process model

IX. Disposability
Maximize robustness with fast startup and graceful shutdown

X. Dev/prod parity
Keep development, staging, and production as similar as possible

XI. Logs
Treat logs as event streams

XII. Admin processes
Run admin/management tasks as one-off processes

Link: https://12factor.net/

The Illustrated Children Guide to Kubernetes

devops,

Explaining Docker and Kubernetes to someone not familiar with all the technical details of container technology can be very frustrating from time to time.

At this point the video by Matt Bucher will come in handy.

Also as a little side effect, this video gives you some hints on how to explain something to… middle and upper management.

Entrypoint Pitfalls in the Mac-o-Windolinux Docker World

devops, , ,

I do some work on my MacBook with macOS, on my Windows laptop with Windows 10 and Ubuntu WSL. I work in Visual Studio Code onWindows while running Ansible scripts in my Ubuntu WSL on the same code base. What could possibly go wrong? While I spent the last few evenings debugging, I completely forgot about the obvious. I should know better, though.

CRLF vs LF

When working on my Windows machine I regular forget about file formats. While in many cases the systems are nowadays very resilient, when creating Docker containers this can end up in a big FUBR. In the likely case, you freshly built container using an entrypoint script tells you during a docker-compose up something like 

standard_init_linux.go:xxx: exec user process caused "no such file or directory"

go and check the file format of the entrypoint script and switch to LF. At least Visual Studio Code makes it easy.

To bash or not to bash

In case you see exactly the same error, check the entrypoint script again. Is it using bash as mine?


Go ahead and make sure bash is installed in your image. Use something like the line below. On a very regular base, I completely forget about installing bash but keep trying to use it again and again.


apk add bash

No Permission

In case you encounter another obscure message telling you 

standard_init_linux.go:xxx: exec user process caused "permission denied"

check the permissions of the entrypoint script. 

chmod +x entrypoint.sh

should do it on on your host. As I run my deployment using Ansible, I use a task similar to

 
- name: Copy entrypoint.sh file
  copy:
    src: entrypoint.sh
    dest: "{{ install_dir }}/entrypoint.sh"
    owner: root
    group: root
    mode: 0755
    force: yes

I am still not sure if setting755 and root are best practices and should be modified.

Proper Logwatch Configuration using Ansible

devops, , ,

On my way setting up a proper monitoring for my server, I just installed Logwatch to receive a daily summary of the what happened recently on the machine. I will have a look into Prometheus, Grafana, Icinga etc. later. However, for now I just wanted a quick summary of the daily “what’s going on on the machine”. Eventually, I had to fix an occur No such file or directory error.

Therefore, I decided to use Logwatch as a lightweight solution to my needs.

Installation Script

The Ansible script to install Logwatch is straight forward:

- name: Install logwatch
  apt:
    name: logwatch
    state: latest 
  tags:
    - logwatch

- name: Create logwatch.conf file for customisations
  file:
    path: /etc/logwatch/conf/logwatch.conf
    state: touch
  tags:
    - logwatch

- name: E-Mail to 
  lineinfile:
    dest: /etc/logwatch/conf/logwatch.conf
    regexp: "^MailTo ="
    line: "MailTo = {{ logwatch_email }}"
    state: present
  tags:
    - logwatch

- name: Set detail  
  lineinfile:
    dest: /etc/logwatch/conf/logwatch.conf
    regexp: "^Detail ="
    line: "Detail = {{ logwatch_detail }}"
    state: present
  tags:
    - logwatch

Configuration & Troubleshooting

I basically set up two parameters, the e-mail as well as the detail level I want for the report. Important to know is the order Logwatch is applying your configuration settings. Following the recommendations, I did not change anything in the configuration file at 

/usr/share/logwatch/default.conf/logwatch.conf

rather I decided to copy the file to 

/etc/logwatch/conf/

The reason is the order, logwatch is scanning for configuration parameters in the following order. Each step actually overwrites the previous one.

  • /usr/share/logwatch/default.conf/*
  • /etc/logwatch/conf/dist.conf/*
  • /etc/logwatch/conf/*
  • The script command line arguments

Eventually, I ended up in the following error:

/etc/cron.daily/00logwatch:
/var/cache/logwatch No such file or directory at /usr/sbin/logwatch line 634.
run-parts: /etc/cron.daily/00logwatch exited with return code 2

To fix this, avoid copying the original configuration to one of the other places. I did this because I followed some recommendation I received. Instead, I now touch a new configuration file as well as setting the two parameters for MailTo= as well as Detail=. Both are s set using Ansible variables in my scripts. The additional configuration file now looks pretty boring, though:

MailTo = mail@example.org
Detail = Low

You also can provide these parameters when calling the script in the cron job: Using Ansible the modification would look like the following: 

lineinfile: 
  dest: /etc/cron.daily/00logwatch                
  regexp: "^/usr/sbin/logwatch"               
  line: "/usr/sbin/logwatch --output mail --mailto {{ logwatch_email }} --detail {{ logwath_detail }}"               
  state: present 
  create: yes

I decided to change the cron job call as one never can be safe from the file changing during package updates. The same should be valid for the configuration file at its origin place.

tl;dr

Setting up Logwatch using Ansible might cause strange “No file or directory”-errors during the cron job call. This can be avoided by applying additional configuration settings at appropriate configuration locations.

Personal DevOps #3

devops, , ,

While most of the prerequisites are met for my automated server setup I came across some issues when I started with my very first Ansible playbooks.

First Ansible Playbooks

First of all, I wanted to start with a quite simple ping playbook, to ensure the servers are reachable by Ansible.

# Playbook to ping all hosts 
---
- hosts: all
  gather_facts: false
  tasks:
    - ping:

When I run this script I was immediately confronted with the very first error. I really love when such things happen. Nothing can motivate one more than immediate failures like the following.

FAILED! => {
"changed": false,
"module_stderr": "Shared connection to xxx.xxx.xxx.xxx closed.\r\n",
"module_stdout": "/bin/sh: 1: /usr/bin/python: not found\r\n",
"msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
"rc": 127
}

As I started with a minimal Ubuntu 18.04 LTS installation, there is simply no Python 2 installed. However, to run the Ansible tasks on the node, Python is required. I made use of the raw task in Ansible to update the package lists as well as install the package python-minimal. In addition, I added the package python2.7-apt in this bootstraper as it is needed later on. Once Python has been installed the ping playbook worked without any problems.

# Bootstrap playbook to install python 2 and python-apt
# It checks first so no unecessary apt updates are performed
---
- hosts: all
  gather_facts: False
  
  tasks:
  - name: install python 2
    raw: test -e /usr/bin/python || (apt -y update && apt install -y python-minimal)
  - name: install python-apt 
    raw: test -e  /usr/lib/python2.7/dist-packages/apt || (apt install -y python2.7-apt)

For both packages, I test for the corresponding directories on the node to avoid unnecessary updates.

Note: When testing for a directory on the shell the following line became very handy:

> [ -e /usr/lib/python2.7/dist-packages/apt ] && echo "Found" || echo "Not found"

At a second step, I created a maintenance playbook to update and upgrade the packages on my node.

# Playbook to update Ubuntu packages 
---
- hosts: all
  gather_facts: false
  tasks:
  - name: update and upgrade apt packages
    become: true
    apt:
      upgrade: yes
      update_cache: yes
      cache_valid_time: 86400

Before including the pyhton-apt package to the bootstraper, I got the following error when dry running the playbook.

fatal: [xxx.xxx.xxx.xxx]: FAILED! => {"changed": false, "msg": "python-apt must be installed to use check mode. If run normally this module can auto-install it."}

Conclusion

While this is not any rocket science for sure, I now have a few essential scripts to bring my server to a base level I can start working with.

Personal DevOps #2

devops,

Even though, I wanted to do this structured, beginning with this topic is quite a mess. You have to set up everything and pull all pieces together before one can start working properly.

Getting a Server

First of all, I picked up a new root server. I will install Ubuntu 18.04 LTS on this particular one. I found a great offer at Netcup, where I just ordered a new root server with unlimited traffic.

That way, I can start moving bit by bit from my old, handcrafted and home-brewed server to the new one instead of replacing the old server. Once everything works fine, I will migrate the data and change the DNS entries to the new server.

macOS as Ansible Control Machine

When starting with such a project, one should expect problems from the very first moment. For me, it started already when I wanted to install Ansible on macOX. Why I’ve chosen Ansible over Chef and Puppet will be covered later.

There is no native Ansible package for macOS available. Instead, you can use Homebrew to do so. Assuming Homebrew is already installed, the following command should do the job.

> brew install ansible

However, for me, it already ended in some errors:

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools),
missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

As most of the time the latest macOS version Mojave might be the reason, running

> xcode-select --install

the command line tools will be installed as part of Xcode. Once accomplished, the homebrew command should work like a charm. Eventually, Ansible is available on my local laptop which is now my control machine.

Setting up the SSH Access

Usually, I deactivate all password-based logins on a server and allow only RSA-key-based logins. To generate the key, you can follow the instruction on ssh.com.

To upload the key, ssh-copy-id is needed. However, once again this is not available on macOS and you have to install it using

> brew install ssh-copy-id

Now you can upload the key using

> ssh-copy-id -i ~/.ssh/key user@host

So far I have planned this was the only step necessary on the server. All future settings, including deactivating password only access should be set using Ansible scripts.

Setting up a Repository

As a (former) developer, I almost don’t do anything without putting my stuff into a revisioning system. For my Ansible scripts, I decided to set up a private GitHub repository. Although I use GitLab and Subversion at the moment at work as well as running a Subversion server at home, I meanwhile put almost everything int GitHub repositories. Therefore Github comes just in quite handy.

Why Ansible?

For automation, there are several frameworks. One major advantage of Ansible is the fact, only a single so-called control machine is needed. There is no further Ansible infrastructure needed. Once you have installed Ansible as described above you are ready to go. For Puppet, you need again to deal with a server and agents an eventually you stick with running daemons. While this is a feasible approach e.g. to manage my team’s 100 servers at work, this is an overkill for personal use. Same with Chef. That’s the reason, I decided to use Ansible as it seems a quite feasible approach for personal use with little overhead while being an approach with the least attack vectors.

If not being familiar with ansible, I recommend watching the following 14-minute video, which gives you a good overview of Ansible’s capabilities. You might want to skip the last four minutes as it deals with Red Hat’s Ansible Tower which targets enterprises.


Personal DevOps #1

devops

I recently gave a talk, introducing DevOps how to add value by fully automated toolchains. Following this at my daily work, I realized, that my very personal servers are managed overall manually. Every single change is an epic battle. Therefore, I decided to get my hands dirty and to set up a fully automated toolchain for my personal server.

Why!?

I want to learn about the tools, the automation and the processes necessary. Maybe you ask yourself, why one should do this. I simply do this to fetch up with the current technology development. I meet a lot of technical managers who actually have no clue about the technology they are talking about. Personally, I don’t want to be one of those managers.

On the other hand, I have to move to a more recent version of my server’s OS but I don’t want to set up the server again by hand.

About one year ago, I had to re-install the server completely after a configuration mistake, three months ago I had to spend hours and hours to manually clean my WordPress installations due to an infected site.  So there is an actual need as well.

I decided to blog about this adventure, which is especially interesting, as the blog itself has to move at one point to the new server. As usual, I blog to keep notes for myself. However, if you find the articles helpful, feel free to drop me a line, recommendations and tips are warmly welcomed.

Goals

I thought about some goals I want to achieve. Ultimately, I want a fully automated deployment pipeline for at least my main server. To achieve this, I want to

  • upgrade to a new version of my server’s OS
  • apply automation scripts such as Puppet, Chef or Ansible
  • containerize everything – at least Mail and Web-Server have to be deployed independently
  • automatically setup and create my WordPress instances
  • set up a mechanism to easily deploy .NET Core Web APIs in containers
  • create a single place where all “data” lives
  • create a mechanism to backup the data
  • establish monitoring and alerting push changes through the toolchain fully automated
  • run everything through repositories and  a CI/CD pipeline

tl;dr

I am going to replace my server and building a fully automated CI/CD pipeline.