My very own Continuous Integration Server (mvoCI)

2020-01-11T16:30:00+01:00

This article has been deprecated by Golangs Modules.

About a year ago, I changed by website (which you are visiting right about now) from a Wordpress based system to a static website generator called Pelican [pelican]. Since then I feared building that website and that it may become one day not buildable, e.g. because I loose the knowledge on how to build the damn thing. Also deploying new versions of the website would be quite a hassle, as the new HTML-files would have been copied over to the server manually.

But fear not! There are solutions to that, in the form of continuous integration servers. Well... I tried Jenkins [jenkins]. Jenkins is a really powerful Java-software, which can do nearly anything you want. However, it is a Java-software. Once it has got its heap-memory, it will never give it back. This would be swell, if the server running it has enough RAM to spare, however my vServer has only 2 GB. And Jenkins after building my website once (a quite simple website, I might add) ate nearly half of it. Jenkins alone! After an incident, where Jenkins spammed a log-file and also ate the disk-space, I swiftly got rid of it.

There are also other Continuous Integration servers, but most of them are quite overblown and full of features I would never need. A very simple one is flux-ci [flux-ci]. It's written in Python, based on Flask and is an easy to use, straightforward solution to my problem. Looking through the code at the time (about a year back), I wasn't satisfied and began work to modify flux-ci to my hearts content. Eventually I stopped to work on that code, and began programming my very own CI-server (mvoCI).

While it shares the templates, i.e. the optics with flux-ci (I've asked), it is its very own beast, written in Go [golang], based on echo and gorm for object-oriented interface to a relational database.

What can it do?

tl;dr:

build git repositories on user input or repo-push
public repositories (builds can be viewed and artifacts downloaded without auth)
query information about repos and builds, as well as build specific branches and commits via a REST API
deploy e.g. a pelican website by abusing the build-script

I tried to stay on the simple side with mvoCI. It can build your git-repository, either when you hit build, or when someone pushes into the repository, if you have a webhook configured in the git-server of your choice. I personally use gitea [gitea] in conjunction with that.

You can have several users, with their own repositories. You can have repositories, which are public, i.e. their builds can be downloaded without authentication. I recently added a REST-API interface, which can be used to query information about repos and builds, and to build a specific repository, commit and branch. More work needs to be done, so someone could be able to build an accompanying app to mvoCI in the future.

I already use mvoCI to deploy new versions of my website. Essentially I am abusing locality; the user executing mvoCI has write access to the folder of the website. After building the website, its www-folder is copied over to the apache DocumentRoot. Deployment of simple services via this technique seems possible. For deploying services or updates over the boundaries of machines the API could be used; this is part of the work to be done in future.

What is planned?

At the time of writing, the API is a bit bare bones. While querying information is already possible, it is not possible to edit information or delete any objects. Also querying if there are any new builds for a repository or even push-notifications if that is the case it not possible yet.

Webhook-Filters

An idea in a semi-abandoned state is the introduction of filters for webhooks. Git-Servers send an event-type with their webhook, so they can indicate, whether the user pushed into the repository of created a new release or whatever. A idea is extend the quite simple model of a repository, with one single build script, to a repo with a list of filters and attached build scripts. So a repo could "listen" to several events and handle them differently, maybe even building the repo several times for a single webhook event.

A generic filter-interface would also allow to do timed builds on more frequented repositories, think about Firefox, where you would want automatic tests for every commit, but maybe would want a release-build only every night or week. A pattern-based method for keeping builds would fit right into that.

This would indeed make mvoCI more powerful, but also bigger and would require a rewrite of some parts of the code, which I'm not willing to do yet. I've started work on it and may be restarting it in the future, however at the moment, mvoCI is more than capable of handling what I want from it.

How do I get it?

tl;dr

Make sure you have at least the go-toolchain in version 1.12.

#!/bin/bash
git clone https://git.stefannaumann.de/snaums/mvoCI.git
cd mvoCI
export GOPATH=$(pwd)
mkdir -p src/mvoCI

# move the code to the go-dir
shopt -s extglob dotglob
mv !(src) src/mvoCI/
shopt -u dotglob

pushd src/mvoCI
# install build dependencies
make dep

# build and move artifacts to dist
make dist
# cleanup
rm -rf web hook log Makefile main.go mvo.cfg static repo static views core build builds
popd

# cleanup for packaging
mv src/mvoCI/dist/* .
rm -rf src pkg

The long version

I'm currently testing building and packaging mvoCI with my own mvoCI instance. Until I make that public, the script under tl;dr is sufficient to build mvoCI.

The code is hosted on my gitea-instance. Clone the repository and move inside.

Requirements

golang in version 1.12 or higher

If you haven't gotten the go toolchain in the correct version, you can install it from the Linux repositories. Make sure you get at least version 1.12. If you are running Debian 10 (stable), you will not get the correct version. You can get a working toolchain from [golang-dl] in that circumstance or when running Windows or Mac OSX.

Build time requirements

As of time of writing mvoCI depends on the following golang-libraries

github.com/jinzhu/gorm (plus it's dependencies for DB access)
github.com/labstack/echo
golang.org/x/crypto/bcrypt
github.com/foolin/goview
github.com/foolin/goview/supports/echoview-v4

If you don't want to bother, just run make dep, which will install all necessary library dependencies. If you notice any problems, don't hesitate to contact me.

Building mvoCI

Make sure it is in your $GOPATH. Either you move the cloned repository to ~/go/src/mvoCI (which would be the default $GOPATH) or you overwrite the environment variable. Make sure you keep the tree intact, i.e. $GOPATH/src/mvoCI/<repo content>. You will need to redownload the dependencies into the correct $GOPATH.

Change in the directory and run make mvobuild or make dist . Both will build mvoCI, but dist will move all necessary files to the dist folder, so you only need to move that to where you want and can start running the application.

Set-up and Usage

Security

Keep in mind, that mvoCI can execute arbitrary bash-code on your machine. If you are running it on your most valuable server, make sure you know what you are doing and keep locked down, so it cannot destroy any other services your machine provides.

At least execute it as a different user from any other. Create a new user, give it a home, copy the dist folder into that. Then you could use systemd to execute the program with the correct user at startup. You may use this config file for systemd:

[Unit]
Description=mvoCI
After=syslog.target
After=network.target
After=mysqld.service
#After=postgresql.service

[Service]
RestartSec=2s
Type=simple
User=mvo
Group=mvo
WorkingDirectory=/home/mvo
ExecStart=/home/mvo/mvo
Restart=always
Environment=USER=mvo HOME=/home/mvo

[Install]
WantedBy=multi-user.target

Installing

For the first installation you will need to run the mvo executable with the parameter --install. This will set the program in Installation-mode. This allows you to set the database parameters and credentials for the first user. It will create a config file mvo.cfg. After the installation process is done, stop the process and run the program without any parameters for production.

See through the configuration options in mvo.cfg. It's a JSON-representation of the struct in core/config.go. Changing anything will require a restart of the application.

Running

Once it is set-up and running you can login with your credentials, create repositories and new users. You can also LoginTokens, for using the API. An API-call needs a valid token in the mvoCi_session token.

What to do next

That is up to you. I'd appreciate bug-reports and pull requests to be reported at the repository. Every polite feedback is good input, what to improve. This includes feature-requests as well as bug-reports.

References

[flux-ci] https://github.com/NiklasRosenstein/flux-ci
[jenkins] https://jenkins.io
[gitea] https://gitea.io
[golang] https://www.golang.org
[golang-dl] https://golang.org/dl/
[pelican] https://www.getpelican.com
[repository] http://git.stefannaumann.de/snaums/mvoCI

snaums.de - golang