Antoine Jacoutot (ajacoutot@) writes in with a story about how M:tier, the company he works for, uses OpenBSD.
For several years now I’ve been working for M:tier, a very OpenBSD-friendly company, and I think it’s about time that we share about “how” we use OpenBSD in some aspects.
As for “what” we use it for, it’s simple: everything.
As a company we are very dedicated to what we do because we are “forced” to use our operating system of choice and we want our customers to be as happy as we are at using it 🙂
So our paid job is hacking on and deploying, maintaining, supporting… OpenBSD installations. We are also required to hack on things that can be merged back into OpenBSD itself and when it’s not possible, then we change what we did so that it can be. Of course some developments are very specific to what we do and have no place in the project’s CVS tree.
So, amongst other services, we set up and maintain several 100% OpenBSD-based infrastructures (going from the entry site firewall to the secretary’s workstation) and this is what I’m going to talk about here.
As a side note, it is important to know that we are working exclusively for Fortune 500 companies (each operating in totally different and unrelated sectors).
What it means is that:
- We are not setting up systems for small geek-friendly-only companies but for huge ones with a long IT history (some of them are present in >100 countries worldwide). While I cannot reveal any names, it is important to know that OpenBSD can fit in the Big Ones.
- We have to comply to very large and complex technical and legal specifications.
While most people will see it as a useless effort, we think it is very interesting to make a non-mainstream operating system comply with the corporate rules.
The Big Picture
We are currently managing over 600 users in several locations around the world (expecting a large increase before the end of the year).
All these locations are fully running under OpenBSD, that is:
- the firewalls: PF, IPSEC, CARP…
- the infrastructure servers: DNS, DHCP, TFTP, FTP, HTTP, NFS, LDAP, puppetmaster, Kerberos, proxy, print server…
- the desktops (workstations and laptops): The GNOME Desktop and plethora of graphical applications.
95% of the services are redundant (including our puppet distribution setup). Shutting down one server has zero impact on production. Everything is monitored by Zabbix: there is a Zabbix proxy running on each site that sends its data over an IPSEC VPN to our central Zabbix server.
We are also developing a Zabbix mobile client that runs on the iPhone, Android, Blackberry and Windows Mobile 6 so that the sites managers can see the status of their OpenBSD machines from anywhere 😉
There have been countless articles about how OpenBSD can be used as a <insert your infrastructure service>, this is why in this article, I will focus on two usually less covered aspects: how we use puppet to manage these installations and the use of OpenBSD as a Joe User Desktop.
Another aspect of our solution is that we exclusively use signed packages. This has nothing to do with security but we want to make sure the packages used on the machines are all coming from us. There is a nice howto about how to build and use signed package in the OpenBSD FAQ.
Joe User meets OpenBSD
As a regular user, when the IT staff starts migrating your Windows XP workstation to Windows 7, it is very traumatic! But, you accept it because everyone else in the world is more or less doing the same. But imagine if you had to migrate from Windows XP to Mac OS X! Ok you can also accept it because, well you own an iPod like everyone else, so Apple is known to you.
But the real challenge comes from migrating users from a >10 years habit of using Windows and MS Office to an OpenBSD GNOME Desktop with Libre/Openoffice without impacting their daily work, aka production, aka company revenue.
The important aspect of such a migration even before installing the new system is obviously information and … information. A user always feels somewhat punished when his computer work environment changes, so it is very important to explain why it is happening and how it will in the end serve and benefit him or her. We have a tendency to be on the user’s side here, people want to get work done and not learn how computers work.
On the technical aspect, a typical Joe User Desktop is made of the following parts:
- a pre-configured GNOME (mandatory screenshots 1 and 2) environment for the user interface
- Libre/OpenOffice suite for enterprise editing
- OpenNX for accessing Windows Remote Desktop or Terminal services (that is needed because companies use internally developed applications running on Windows only)
- the rest of the graphical applications needed for daily work (web browser, ftp client, multimedia player, picture editor…)
The Desktop configuration in itself is done using a script containing gconf commands (this script is run by the puppet daemon).
We can totally transform the Desktop by running gconftool-2 with correct arguments.
We provide some (really not many) scripts to ease daily usage of the Desktop. After several years of using Windows, people are used to a certain behavior of their computer and we try to bring that to them (e.g. pluging a USB key will automatically mount it on the Desktop thanks to an hotplugd(8) script and open a nautilus window with a right-click menu to unmount the device).
We also wrote some graphical interfaces for commonly used tasks like encrypting a USB drive, restarting the network, setting up ppp/pppoe/WiFi connections, connecting to the office IPSEC VPN, changing the local and Kerberos passwords…
At last, it is good to know that all users’ home directories on the laptop are encrypted using standard OpenBSD tools and with a passphrase that is different for each user.
Everything a user needs is just a click away, he/she is by no means required to use the terminal.
Joe User meets tiny OpenBSD
In some situations, a full-blown Desktop is not required. For this particular case, we have developed an OpenBSD-based thin client solution.
The thin client is a small OpenBSD installation with base and x* sets which runs under a read-only “/” partition with others that are needed for normal operation as mfs filesystems in read-write mode (these are /tmp, /home, /var/log and /var/db).
When the boot process is done, X.org and the thin client software get started right away from /etc/ttys.
The thin client uses a central update server running puppet on the M:tier side, and on the client puppet remounts “/” read-write, does its run then remounts it back to read-only mode.
Depending on the configuration IPSEC may be used. IPSEC and LAN status are all indicated on the client itself.
Note that besides a modified /etc/rc (stripped down to speed up boot times) the thin client is running a standard OpenBSD installation.
Installation, maintenance and configuration
Here we will talk in a bit more details about some technical sides of managing all the different locations.
Installation
The installation process for desktops and servers is the same. On site we have a redundant pxeboot server and puppetmaster that holds all the necessary pieces for an unattended installation.
Installing a machine is just a matter of PXE booting on a modified bsd.rd kernel. This bsd.rd automatically partitions the hard drive and installs the required sets from the local ftp according to configuration files held on the pxeboot server.
At the end of the installation, the ruby-puppet package is installed and puppetd is started. Puppet will then finish the installation using different manifests and recipes whether we are installing a laptop, workstation, server…
Puppet takes care of everything from installing the packages to creating daemon users and configuring the whole setup.
Once the installation is finished, the machine automatically reboots and we use rc.firsttime to do some interactive post-install administrative tasks (like setting up a root password, registering the machine into the Kerberos realm…).
Note that the puppetmaster is running with –servertype=mongrel and we use nginx to load balance the requests which is required to have decent concurrent connections support.
Configuration deployment
For configuration purposes we exclusively use puppet. Amongst the obvious benefit of having a central location for configuration files, it also allows us to setup generic configuration files that will be shared amongst all customers but that we can still make specific by only changing some variables.
The puppet recipes and manifests are actually contained into a package. When a configuration change is requested, we edit the recipe, create the new package which then gets pulled by the requesting site and the puppetmaster does the rest.
Obviously, most of the tasks are automated with sanity checks but an interactive manual approval is always needed before deploying a new configuration.
Being written in Ruby, puppet is easily extensible and can be made to do pretty much anything. We are abusing this feature quite a lot 🙂
Maintenance
Surprisingly, we are also using puppet for that 😉 The puppet daemon running on all machines will automatically check for packages updates, new antivirus definitions… Yes we use an antivirus because we are interacting with Windows users and the private and public shares that are available to the users over NFS are also available via FTP (encapsulated into the IPSEC VPN) from the remote Windows Terminal server.
For base system updates, we use “binpatches” built using the binpatch-ng framework.
A binpatch is just a regular OpenBSD package containing the updated binaries that also creates a backup of the replaced files at pkg_add(1) time so that the update can be reverted in case something goes wrong. Binpatches are handled using the regular OpenBSD package tools.
Backups are handled in two different ways. On the Desktops we use “backintime” which takes an incremental snapshot every hour into a hidden private share on the NFS server. Backintime is really handy for users because it provides an easy to use graphical interface to restore files and or directories from a particular point in time. For enterprise compliant backup, we use bacula to backup all the servers to a central point (that includes the NFS server, so in fact user data has a double backup). For bare metal recovery, all that is needed is to bootstrap a server from scratch then running bacula to restore everything. Critical files are of course saved off site as well. This dual setup allows for easy access to backup recovery mode while maintaining compliancy with corporate specifications.
Obviously not being physically on site, we need to have local system administrators. Almost none of them ever heard of OpenBSD before, some exclusively worked on Windows and know nothing else. To deal with that, we provide them with several days of theoretical and practical trainings which are of HUGE interest for us because we get questions and feedback that we would never get from people used to using Unix systems.
Conclusion
It usually takes several months for a complete office to finally be accustomed to its new solution but in the end users are very happy with their new work environment.
It is a great pride for us when entering an office and seeing all these users running OpenBSD on their machine. While from a first glance it just looks like another office like we’ve all seen countless times, when you actually pay attention at what is on the screen it is quite different 😉
While we are facing several social, legal and technological challenges, one of the most interesting one is the use of OpenBSD on the desktop. Most people use OpenBSD as a server, router, firewall or embedded device but for a desktop it is pretty rare (I’m not accounting OpenBSD enthusiasts that are the only ones running an OpenBSD workstation at their workplace). That means we are building our experience from scratch because there is not much feedback from anywhere else.
The best of all is that there is no clear line between our hobby and our work!
I hope you enjoyed reading this article that I hope uncovered some ideas about how OpenBSD can be used in the enterprise world in a way that is not obviously expected.
