My very own Continuous Integration Server (mvoCI)
About a year ago, I changed by website (which you are visiting right about now) from a Wordpress based system to a static website generator called Pelican [pelican]. Since then I feared building that website and that it may become one day not buildable, e.g. because I loose the knowledge on how to build the damn thing. Also deploying new versions of the website would be quite a hassle, as the new HTML-files would have been copied over to the server manually.
But fear not! There are solutions to that, in the form of continuous integration servers. Well... I tried Jenkins [jenkins]. Jenkins is a really powerful Java-software, which can do nearly anything you want. However, it is a Java-software. Once it has got its heap-memory, it will never give it back. This would be swell, if the server running it has enough RAM to spare, however my vServer has only 2 GB. And Jenkins after building my website once (a quite simple website, I might add) ate nearly half of it. Jenkins alone! After an incident, where Jenkins spammed a log-file and also ate the disk-space, I swiftly got rid of it.
There are also other Continuous Integration servers, but most of them are quite overblown and full of features I would never need. A very simple one is flux-ci [flux-ci]. It's written in Python, based on Flask and is an easy to use, straightforward solution to my problem. Looking through the code at the time (about a year back), I wasn't satisfied and began work to modify flux-ci to my hearts content. Eventually I stopped to work on that code, and began programming my very own CI-server (mvoCI).
While it shares the templates, i.e. the optics with flux-ci (I've asked), it is its very own beast, written in Go [golang], based on echo and gorm for object-oriented interface to a relational database.
What can it do?
- build git repositories on user input or repo-push
- public repositories (builds can be viewed and artifacts downloaded without auth)
- query information about repos and builds, as well as build specific branches and commits via a REST API
- deploy e.g. a pelican website by abusing the build-script
I tried to stay on the simple side with mvoCI. It can build your git-repository, either when you hit build, or when someone pushes into the repository, if you have a webhook configured in the git-server of your choice. I personally use gitea [gitea] in conjunction with that.
You can have several users, with their own repositories. You can have repositories, which are public, i.e. their builds can be downloaded without authentication. I recently added a REST-API interface, which can be used to query information about repos and builds, and to build a specific repository, commit and branch. More work needs to be done, so someone could be able to build an accompanying app to mvoCI in the future.
I already use mvoCI to deploy new versions of my website. Essentially I am abusing locality; the user executing mvoCI has write access to the folder of the website. After building the website, its www-folder is copied over to the apache DocumentRoot. Deployment of simple services via this technique seems possible. For deploying services or updates over the boundaries of machines the API could be used; this is part of the work to be done in future.
What is planned?
At the time of writing, the API is a bit bare bones. While querying information is already possible, it is not possible to edit information or delete any objects. Also querying if there are any new builds for a repository or even push-notifications if that is the case it not possible yet.
An idea in a semi-abandoned state is the introduction of filters for webhooks. Git-Servers send an event-type with their webhook, so they can indicate, whether the user pushed into the repository of created a new release or whatever. A idea is extend the quite simple model of a repository, with one single build script, to a repo with a list of filters and attached build scripts. So a repo could "listen" to several events and handle them differently, maybe even building the repo several times for a single webhook event.
A generic filter-interface would also allow to do timed builds on more frequented repositories, think about Firefox, where you would want automatic tests for every commit, but maybe would want a release-build only every night or week. A pattern-based method for keeping builds would fit right into that.
This would indeed make mvoCI more powerful, but also bigger and would require a rewrite of some parts of the code, which I'm not willing to do yet. I've started work on it and may be restarting it in the future, however at the moment, mvoCI is more than capable of handling what I want from it.
How do I get it?
Make sure you have at least the go-toolchain in version 1.12.
#!/bin/bash git clone https://git.stefannaumann.de/snaums/mvoCI.git cd mvoCI export GOPATH=$(pwd) mkdir -p src/mvoCI # move the code to the go-dir shopt -s extglob dotglob mv !(src) src/mvoCI/ shopt -u dotglob pushd src/mvoCI # install build dependencies make dep # build and move artifacts to dist make dist # cleanup rm -rf web hook log Makefile main.go mvo.cfg static repo static views core build builds popd # cleanup for packaging mv src/mvoCI/dist/* . rm -rf src pkg
The long version
I'm currently testing building and packaging mvoCI with my own mvoCI instance. Until I make that public, the script under tl;dr is sufficient to build mvoCI.
The code is hosted on my gitea-instance. Clone the repository and move inside.
- golang in version 1.12 or higher
If you haven't gotten the go toolchain in the correct version, you can install it from the Linux repositories. Make sure you get at least version 1.12. If you are running Debian 10 (stable), you will not get the correct version. You can get a working toolchain from [golang-dl] in that circumstance or when running Windows or Mac OSX.
Build time requirements
As of time of writing mvoCI depends on the following golang-libraries
- github.com/jinzhu/gorm (plus it's dependencies for DB access)
If you don't want to bother, just run
make dep, which will install all necessary library dependencies. If you notice any problems, don't hesitate to contact me.
Make sure it is in your
$GOPATH. Either you move the cloned repository to
~/go/src/mvoCI (which would be the default
$GOPATH) or you overwrite the environment variable. Make sure you keep the tree intact, i.e.
$GOPATH/src/mvoCI/<repo content>. You will need to redownload the dependencies into the correct
Change in the directory and run
make mvobuild or
make dist . Both will build mvoCI, but
dist will move all necessary files to the
dist folder, so you only need to move that to where you want and can start running the application.
Set-up and Usage
Keep in mind, that mvoCI can execute arbitrary bash-code on your machine. If you are running it on your most valuable server, make sure you know what you are doing and keep locked down, so it cannot destroy any other services your machine provides.
At least execute it as a different user from any other. Create a new user, give it a home, copy the
dist folder into that. Then you could use systemd to execute the program with the correct user at startup. You may use this config file for systemd:
[Unit] Description=mvoCI After=syslog.target After=network.target After=mysqld.service #After=postgresql.service [Service] RestartSec=2s Type=simple User=mvo Group=mvo WorkingDirectory=/home/mvo ExecStart=/home/mvo/mvo Restart=always Environment=USER=mvo HOME=/home/mvo [Install] WantedBy=multi-user.target
For the first installation you will need to run the
mvo executable with the parameter
--install. This will set the program in Installation-mode. This allows you to set the database parameters and credentials for the first user. It will create a config file
mvo.cfg. After the installation process is done, stop the process and run the program without any parameters for production.
See through the configuration options in
mvo.cfg. It's a JSON-representation of the struct in
core/config.go. Changing anything will require a restart of the application.
Once it is set-up and running you can login with your credentials, create repositories and new users. You can also LoginTokens, for using the API. An API-call needs a valid token in the
What to do next
That is up to you. I'd appreciate bug-reports and pull requests to be reported at the repository. Every polite feedback is good input, what to improve. This includes feature-requests as well as bug-reports.