04 Jul 2017, 07:26

Problem: I am a human being

Solution: I don’t have one. It’s not a real problem.

Yesterday, I saw someone asserting that open source projects have poor UX, and that this is the result of a toxic culture created by developers who both don’t care about users and are incapable of making good tools. I feel this was a very broad assertion to make. I also feel that making demands upon what people produce for free within a society that makes selling labor for money a prerequisite for survival shows a certain level of entitlement. While I can understand frustration with the lack of polish many open source tools display, attributing this lack of polish to malice shows a distinct lack of empathy for people who are often well aware of the faults of their projects, and struggling to find time and help.

I write open source software. My contributions are to various tools that are relatively obscure unless you have some specialized interests. I’ve contributed to libaries for load testing, distributed computing, and log aggregation. I’ve started two relatively real projects - a Go wrapper for the ZeroMQ distributed messaging library, and a Go syslog message parser. I’m relatively well known in a community or two, and I’ve given a couple of conference talks and spoken at a local meetup or two, but I am not a “name brand” developer.

If I examine why I spend some considerable amount of my time producing software for free within an economic system based on the concept of selling labor for money, I find a multitude of reasons.

One reason is that free software changed my life. I don’t have a college degree - I’m a self taught software developer. I started my post highschool life washing dishes for a living. Finding the book “The Linux Bible” in a book store in the mid 90s changed the course of my life dramatically. It came with a CD with a copy of Yggdrasil Plug and Play Linux, and all the source code. This was when free software outside of very specific circles was still unheard of. If you are a person who has come of age during the time of ubiquitous internet access and github, you cannot know what having access to the source code of an entire operating system meant in the mid 90s. When I saw the impact this access had on my own life in the coming years, I began to view closed source software as unethical. In a society that was increasingly mediated by software, restricting access to learning about software works is in my opinion a means of control and subjugation.

Another reason is that I simply enjoy writing code as means to explore ideas, make things, communicate and share with other people. Creating free software on my own time allows me to work and share outside of the constraints of business value that must be considered when you are trading your expertise for money from a company. It allows space for creating software from a different set of values than those that come with solving problems people are paying you to solve. A far simpler and less wordy explanation is: it’s fun!

A third reason is the people. I’ve made wonderful life long friends working on open source. People who I clicked with instantly, who shared many of my own interests.. people not at all like me, who have challenged my thinking and exposed me to perspectives other than my own. People who have mentored me, and people I have mentored. I have spent most of my life in the same city.. having friends and colleagues all over the world was an unthinkable thought to me in my 20s, and now I’m discussing traveling to Brussels for FOSDEM in January and getting excited about reuniting with my overseas friends.

If I have a point here outside of reminiscing a bit, it’s that the people giving their free time to produce open source software are people, each with their own story. I live in a mid-sized city in the U.S. with my family. As I write this blog post I’m also chatting with my wonderful wife and best friend of 25 years, who is on her laptop researching techniques for working with children with autism. I have a son who is upstairs asleep, and most definitely stayed up too late playing on the computer last night. My daughter (currently in college) is off at the beach with her boyfriend. Our dogs and cats are all scattered around the room napping. Hi! I’m a real person, and not just a robot that produces software for free. Any time I spend on open source is time that I most assuredly could and probably should be spending on something else.

There is no such thing as “open source developers”. There are people, who write open source software, along with all the other things they do. When you are frustrated that some software doesn’t do what you need, or is hard to use, or has confusing or incomplete documentation, please remember this. Instead of inferring that the project maintainers are malicious elitests who don’t care about other people, consider another possibility: that they are generous people giving away their work to benefit others, but that they don’t have unlimited time to spend on it. Consider that people prioritizing other things in their life is ok. Consider that being angry that other people aren’t prioritizing what you want with their own time is perhaps being a bit entitled.

04 Mar 2017, 14:43

Problem: I want to use Go service discovery clients with CZMQ

Solution: zdiscgo

Background

At work, I have a tier of Rsyslog instances on an Apache Mesos cluster for two purposes: indexing logs into Elasticsearch via the Rsyslog omelasticsearch plugin, and publishing logs on ZeroMQ endpoints for people who wish to subscribe to logs from specific hosts and services. The rsyslog instances are scheduled to run on arbitrary hosts, and the ZeroMQ sockets are bound to random ports on the hosts the instances end up on. The ports themselves are labeled, so that they can be searched for using the scheduler’s task API.

The first client I used with these endpoints was a command line client that I wrote in Go, using GoCZMQ. It was fairly painless to use the Marathon scheduler’s API (Marathon is a scheduler that works with Mesos) via the go-marathon library to look up the current hosts and ports the ZeroMQ endpoints are exposed on and use that information to connect to them.

Recently, I started on another client written in straight C. I wanted to use the liblognorm log normalization library in this client, and did not want to deal with the overhead of using cgo to call the library from Go. That being said, I also did not want to go through the pain of writing marathon API calls in straight C - making HTTP requests and parsing JSON returns would be a lot of code.

As I was puzzling over which way I wanted to solve this problem, I stumbled on a great blog post from Vladmir Vivien, “Calling Go Functions from Other Languages”. Since I had plans to start experimenting with ZeroMQ + Kubernetes in addition to solving this specific problem, I decided to solve the problem of “calling Go service discovery libraries from C” in a general sense.

The end result is zdiscgo, a czmq zactor implementation that can load Go libraries at run time for service discovery. Let’s take a look at how this works!

Problem: I need a Go interface for my shared library

First I designed a minimal interface for the Go code. I’m using the term “interface” somewhat loosely here, as C does not know about Go interfaces and will just directly load the function. ZDiscgoDiscoverEndpoints(url, key string) *C.char seemed like enough to get started with. It accepts a URL for a service discovery API endpoint, and a key. It makes the assumption that they key will be used in some fashion to look up ZeroMQ endpoints via an API call to the service at the URL, and that a comma delimited list of ZeroMQ endpoints will be returned. The CZMQ ZeroMQ API has constructor functions for zsock_t * (a handle to a ZeroMQ socket) that can accepted a comma delimited list of endpoints, so the result can be passed directly to those functions.

Since C functions can only return one result, the function is expected to handle an error by simply returning an empty string. In the interest of not having to deal with passing references to Go data structures to C, the function is expected to return a *C.char. Since you need to import C in order to export a function to C anyway, I feel that this isn’t too onerous.

For testing, the zdisgco repo includes libmockdiscgo.go, a mock implementation that combines the passed in url and key and returns them:

package main

import "C"
import "fmt"

//export ZDiscgoDiscoverEndpoints
func ZDiscgoDiscoverEndpoints(url, key string) *C.char {
	return fmt.Sprintf("inproc://%s-%s", url, key)
}

func main() {}

This code can then be compiled with -buildmode=c-shared into a shared library that can be used from C:

go build -o libmockdiscgo.so -buildmode=c-shared libmockdiscgo.go 

The above will create both the libmockdiscgo.so shared library, and a C header. The next step was to write a minimal API to make using this from C as pleasant as possible.

Problem: I need a friendly C API to wrap my Go library

Now that I had the Go library worked out, I needed a C API for using it. zdiscgoplugin.c provides a simple C API for loading Go plugins that are written to the above specification. It is written in accordance with 21/CLASS, the style guide that is typically used for ZeroMQ organization projects. The selftest provides a good overview of how to use it:

    //  @selftest
    zdiscgoplugin_t *self = zdiscgoplugin_new ("./go/libmockdiscgo.so");
    assert (self);

    const char *endpoints = zdiscgoplugin_discover_endpoints (self, "url", "key");
    assert (streq ("inproc://url-key", endpoints));

    zdiscgoplugin_destroy (&self);
    //  @end
    printf ("OK\n");:1

Let’s take a look at how it works, starting with the zdiscgoplugin_t struct. This struct holds a handle to the dynamically loaded library, and a reference to our Go ZDiscgoDiscoverEndpoints function:

struct _zdiscgoplugin_t {
    void *handle;
    char * (*discover)(go_str, go_str);
};

The constructor accepts the path to a dynamic library and returns a pointer to the struct:

zdiscgoplugin_t *
zdiscgoplugin_new (char *libpath)

The first thing the constructor does is allocate memory for the struct. If there is an error allocating this memory, it returns a NULL. Unlike Go, C only supports a single return value for a function. A caller of this function is responsible for checking that the return is not NULL.

    zdiscgoplugin_t *self = (zdiscgoplugin_t *) zmalloc (sizeof (zdiscgoplugin_t));
    if (!self)
        return NULL;

dlopen loads the shared library file and returns a handle to the loaded object. On Linux this function is part of dlfcn.h, which defines functions for working with shared libraries.

    self->handle = dlopen (libpath, RTLD_NOW);

dlsym accepts a handle to a shared library and the name of symbol in that library, and returns the memory address of the symbol, which is then stored in the struct as a function pointer:

#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-pedantic"
    self->discover = (char * (*)(go_str, go_str)) dlsym(self->handle, "DiscoverEndpoints");
#pragma GCC diagnostic pop 

    return self;
}

But what is all this #pragma stuff? As it turns out, dlsym returns an object pointer, and ISO C does not allow the conversion of an object pointer to a function pointer. Compiling this code with warnings enabled will result in a warning:

src/zdiscgoplugin.c:46:22: error: ISO C forbids conversion of object pointer to function pointer type [-Werror=pedantic]

I used a pragma to turn off pedantic warnings for this line of code.

The zdiscgo_discover_endpoints call accepts a zdiscgoplugin_t *, and two char *s (an url to a service discovery service, and the key to use for a lookup):

const char *
zdiscgoplugin_discover_endpoints (zdiscgoplugin_t *self, char *url, char *key)

The first thing the function does is convert the url and key to a go_str, which is a custom C type that is memory compatible with the Go string type. This type is defined in zdiscgoplugin.h:

typedef long long go_int;
typedef struct{const char *p; go_int len;} go_str;

The conversion itself is relatively straightforward:

    go_str discover_url = {url, strlen (url)};
    go_str discover_key = {key, strlen (key)};

Now, we can call the go function through reference held in self->discover:

    char *endpoints = self->discover (discover_url, discover_key);
    return endpoints;

The zdiscgoplugin_destroy function cleans up our memory when we are finished. In this case it does not really do much, but it makes for a nice complete API and will be handy in the future as we expand on things:

void
zdiscgoplugin_destroy (zdiscgoplugin_t **self_p)
{
    if (*self_p) {
        zdiscgoplugin_t *self = *self_p;
        //  Free class properties here
        //  Free object itself
        free (self);
        *self_p = NULL;
    }
}

Problem: I don’t want service discovery calls to block my application.

Making calls over the network to service discovery systems can take quite a bit of time. In order to allow asyncronous requests and replies, I wanted the ability to run the service discovery interface in its own thread. The CZMQ library provides a class for exactly this purpose: zactor. A zactor is similar in concept to a microservice, except it runs as a thread in your local process. Like a microservice, all communication to and from the thread is via protocol messages (as opposed to sharing memory between threads). Since ZeroMQ uses the same socket API for communicating between threads as it does for communicating over network transports, the code looks the same as it would if you were talking to a remove service.

The first thing I did was design a small command protocol for the actor. Each command in the command protocol is a command frame followed by 1 or more argument frames. These commands are handled internally by zdiscgo_recv_api.

First, the handler receives the full message in the form of a zmsg_t *. It pops the first frame off the message and checks the command.

The VERBOSE command sets verbose mode for the service. Debugging information will be sent to stdout. The command handling code simply sets a verbose flag in the zdiscgo_t struct.

The CONFIGURE command is a 2 frame command. The second frame contains the path to the Go shared library to load. The command handling code attempts to construct a zdiscgoplugin_t * using the supplied path. It returns a 0 on success and a -1 if there is an error.

The DISCOVER command is a 3 frame command. The second and third frames are the url and key used by ZDiscgoDiscoverEndpoints. It will call our Go library, and sends back a single frame containing a comma delimited list of endpoints on success or an empty string on failure.

These details are abstracted away from the user’s perspective. You don’t have to worry about dealing with threads or inter-thread communication. You just make a new instance of the class and use it.

    //  Create a zdiscgo instance. This will spin up a new OS level
    //  thread that will handle service discovery requests.

    zactor_t *zdiscgo = zactor_new (zdiscgo_actor, NULL);

    //  We communicate with the service discovery thread over
    //  a ZMQ_PAIR socket. You can pass the zdisgco instance
    //  to any CZMQ methods that accept zsock_t *. 
    //  Let's set the service to verbose mode.

    zdiscgo_verbose (zdiscgo);

    //  Next, let's configure the service by telling it to load 
    //  our go shared library. The zstr_sendx command will send
    //  multiple string frames. A NULL terminates the message.
    
    int rc = zdiscgo_load_plugin (zdiscgo, "./go/libmockdiscgo.so");
    assert (rc == 0);

    //  Now let's get some endpoints! We send a DISCOVER command
    //  that consists of the url of a service discovery service,
    //  and the identifer the service should use to find the 
    //  endpoints we want.

    char *endpoints = zdiscgo_discover (zdiscgo, "url", "key");

    //  Check that we have the correct response

    assert (streq (endpoints, "inproc://url-key"));

    //  Shut down the zdisgco instance and clean up memory.

    zactor_destroy (&zdiscgo);

Let’s take a deeper look at how all of this works, starting with zdiscgo_t

struct _zdiscgo_t {
    zsock_t *pipe;              //  Actor command pipe
    zpoller_t *poller;          //  Socket poller
    zdiscgoplugin_t *plugin;    //  zdiscgoplugin_t reference
    bool terminated;            //  Did caller ask us to quit?
    bool verbose;               //  Verbose logging enabled?
};
  • zsock_t *pipe holds the zeromq socket that is used to send and receive messages between threads. If you are a Go programmer, you can think of this as somewhat similar to a Go channel.
  • zsock_t *poller holds the poller that is used to watch for incoming messages on the socket.
  • zdiscoplugin_t *plugin holds the plugin, which manages communicating with the Go shared libary.
  • bool terminated is set to true when zactor_destroy is called and triggers shutdown / cleanup.
  • bool verbose is initially set to false, and is set to true by a call to zdiscgo_verbose.

Next up is the zactor function itself. The zactor constructor expects a function of type void (zactor_fn) (zsock_t *pipe, void *args):

    zactor_t *zdiscgo = zactor_new (zdiscgo_actor, NULL);

Here is my implementation of the function:

void
zdiscgo_actor (zsock_t *pipe, void *args)
{
    zdiscgo_t * self = zdiscgo_new (pipe, args);
    if (!self)
        return;          //  Interrupted

    //  Signal actor successfully initiated
    zsock_signal (self->pipe, 0);

    while (!self->terminated) {
        zsock_t *which = (zsock_t *) zpoller_wait (self->poller, 0);
        if (which == self->pipe)
            zdiscgo_recv_api (self);
       //  Add other sockets when you need them.
    }
    zdiscgo_destroy (&self);
}

This function constructs an instance of zdiscgo_t and then goes into a while loop. It waits for command messages from another thread, and passes those commands to zdiscgo_recv_api, which contains the callbacks for each command. Let’s take a look at what happens behind the scenes of the int rc = zdiscgo_load_plugin (zdiscgo, "./go/libmockdiscgo.so") call:

int 
zdiscgo_load_plugin (zactor_t *self, char *path) {
    zstr_sendx (self, "CONFIGURE", path, NULL);
    int rc;
    zsock_recv (self, "i", &rc);
    return rc;
}

zstr_sendx can accept either a zsock_t * or a zactor_t * as an argument, and a variadic list of strings, that it will send as a multi-frame ZeroMQ message (one frame per argument). NULL being passed as the last argument tells the function there are no more frames.

As explained above, the zdiscgo_actor implementation running in another thread will pass the receive socket to the zdiscgo_recv_api for handling. Here is the code which handles the CONFIGURE command:

    if (streq (command, "CONFIGURE")) {
        if (self->verbose)
            zsys_debug ("received 'CONFIGURE' command");

        char *libpath = zmsg_popstr (request);
        int rc; 
        self->plugin = zdiscgoplugin_new (libpath);
        if (self->plugin) {
            rc = 0;
            if (self->verbose)
                zsys_debug ("loaded plugin: '%s'", libpath);
        } 
        else {
            rc = -1;
            if (self->verbose)
                zsys_error ("could not load plugin: '%s'", libpath);
        }

        zsock_send (self->pipe, "i", rc);
    }
  1. The path argument is popped out of the received message.
  2. It is passed to zdiscgoplugin_new, which is responsible for loading the go plugin.
  3. A reply message is sent, containing a single frame with either a 0 on success or -1 on failure.

The rest of the commands are implemented in a similar fashion.

Next Steps

I’m currently using this library in a tool at work that uses Marathon service discovery as described at the start of this article. Here’s a snippet of code showing it in use:

    zactor_t *zdiscgo = zactor_new (zdiscgo_actor, NULL);
    zdiscgo_verbose (zdiscgo);
    rc = zdiscgo_load_plugin (zdiscgo, lib);
    if (rc != 0)
        return 1;

    char *endpoints = zdiscgo_discover (zdiscgo, "url", "key");
    if (verbose)
        zsys_debug ("attaching socket to %s", endpoints);
 
    // connect the socket
    rc = zsock_attach (client, endpoints, false);
    if (rc == -1) {
        zsys_error ("zsock_attach failed");
        zsock_destroy (&client);
        return 1;
    }

I hope to soon experiment with a couple of other service discovery systems, and tweak things as I find usability issues and better ways to do things. I’ve released the code to the ZeroMQ organization under the Mozilla Public License. While the problem it solves is fairly esoteric, if it solves a problem for you, please let me know, and feel free to contribute!

09 Oct 2016, 08:56

Problem: My Friend Pieter is Dead

Solution: Carry his ideas forward.

It was March of 2016 when I got the news from Pieter that his cancer was back, and that it was terminal this time. Shortly after that in April, he wrote “A Protocol for Dying” and announced it to the world. He lived much longer than was first expected, dying on October 4th on “his own terms” once cancer made it impossible for him to live on his own terms. I am left with grief, wonderful memories, and membership in an amazing community of software developers that Pieter built.

I want to write some about the ideals that Pieter held that I also share - most importantly I think, his fierce dedication to taking power away from gatekeepers, and his belief in the power of collective intelligence. Right now, I’m still too sad and tired to muster up the energy required - but I felt the need to write the bookend to my April post.

If you want to know what Pieter stood for, read his books. You can find them for free on the internet, but if you can afford to buy them please do - the money from the sales will help support his three children.

30 Apr 2016, 22:35

Problem: My Friend Pieter is Dying

Solution: There’s no solution.

My friend Pieter Hintjens is dying. I’m going to miss him. It’s a truth in life that sometimes we are presented with problems that cannot be solved. This is one of them. Pieter is now facing the ultimate unsolvable problem - the one that we will all eventually face. I’m facing the problem of navigating a storming sea of thoughts and emotions that I’m experiencing as a person who knows him, and values his presence in my life, and is facing losing that presence. None of these thoughts and emotions really fit into words for me. This is a constant and different problem in my life - words always feel too small to me and stringing them together into sentences is so linear, slow, and inadequate.

Pieter, I’ll always carry you with me. All of the colors and textures of the too few moments we got to spend together. Stumbling drunk through London within hours of conversation and laughter. Discussing ideas for ZeroMQ and thoughts about psycopathy in Brussels. Having the pleasure of introducing you to various friends. Music in the warehouse.

It was good, my friend. Every moment was good. Thank you for the moments shared. All of them are kept, and treasured. Your gestures will carry their momentum beyond you through those who understand what you were trying to say with ZeroMQ, and we’ll keep showing others.

It has been an honor and a rare pleasure.

26 Sep 2015, 16:57

Problem: I want to use ZeroMQ with Go

Solution: GoCZMQ

Background

ZeroMQ is a distributed messaging library focused on efficiency and simplicity. CZMQ is a high level C binding for ZeroMQ. It provides a clean API across multiple versions of ZeroMQ that additionally provides a suite of useful services. I have been using ZeroMQ and CZMQ in projects for several years and occassionally contributing to CZMQ and other ZeroMQ related projects. When I first started experimenting with Go, I felt its focus on simple concurrency was a good match with the guiding philosophy behind ZeroMQ.

While Pebbe’s Go bindings for ZeroMQ were available at the time and looked to be high quality, I decided writing a Go language binding for CZMQ would be a fun way to learn about Go’s cgo interface to C.

I decided to develop the bindings as a ZeroMQ organization project using the Collective Code Construction Contract process. While I had contributed to ZeroMQ projects using this process, I had never started a project from scratch using it. I started the project with the pull request “Problem: there is no documentation” on September 6th, 2014.

Since then, I picked up a regular collaborator (hi Luna!) and five other other contributors. The API is stable, and myself and others are building projects on top of it. The experience has been great, and there are many things I could discuss about what I’ve learned on the way - but for now, let’s just answer the question “how do I use it?”.

Installation

These instructions are for building GoCZMQ from git master. We’ll build it against CZMQ from git master, which is in turn built against ZeroMQ from git master. If you’re used to stable releases, this may seem odd. In ZeroMQ organization projects, we do not use development branches. We also don’t care much about traditional “stable” releases. Version tags are mostly to make packaging easier for OS maintainers who care about such things. This practice is codified in the C4.1 process documentation:

“The project SHALL have one branch (“master”) that always holds the latest in-progress version and SHOULD always build.”

“The project SHALL NOT use topic branches for any reason. Personal forks MAY use topic branches.”

libsodium

First, we need libsodium. Sodium is a “modern, easy-to-use software library for encryption, decryption, signatures, password hashing and more”. ZeroMQ relies on it for encryption. It is likely there are dev packages for it for your OS, but here are the instructions for building it just in case:

wget https://download.libsodium.org/libsodium/releases/libsodium-1.0.3.tar.gz
wget https://download.libsodium.org/libsodium/releases/libsodium-1.0.3.tar.gz.sig
wget https://download.libsodium.org/jedi.gpg.asc
gpg --import jedi.gpg.asc
gpg --verify libsodium-1.0.3.tar.gz.sig libsodium-1.0.3.tar.gz
tar zxvf libsodium-1.0.3.tar.gz
cd libsodium-1.0.3
./configure; make check
sudo make install
sudo ldconfig
zeromq

Next up, we build ZeroMQ with libsodium support:

git clone git@github.com:zeromq/libzmq.git
cd libzmq
./autogen.sh
./configure --with-libsodium
make check
sudo make install
sudo ldconfig
czmq

Now, we’ll build CZMQ against ZeroMQ. For an overview of what the CZMQ API provides, see the reference manual.

git clone git@github.com/zeromq/czmq.git
cd czmq
./autogen.sh
./configure
make check
sudo make install
sudo ldconfig
goczmq

Now, finally we can build GoCZMQ itself:

mkdir -p $GOPATH/src/github.com/zeromq
cd $GOPATH/src/github.com/zeromq
git clone git@github.com/zeromq/goczmq.git
cd goczmq
go test -v
go install

For an overview of the GoCZMQ API, see the godoc.org

Hello World

No “getting started” post is complete without the requisite “hello world” example. In this example, we’ll connect a “push” socket to a “pull” socket in the same process, and send a message. This is probably not something we’d do in a real use case, but it’s a nice demonstration of how ZeroMQ is asyncronous.

helloworld.go
package main

import (
	"fmt"

	"github.com/zeromq/goczmq"
)

func main() {
	push, err := goczmq.NewPush("tcp://127.0.0.1:31337")
	if err != nil {
		panic(err)
	}

	pull, err := goczmq.NewPull("tcp://127.0.0.1:31337")
	if err != nil {
		panic(err)
	}

	err = push.SendFrame([]byte("Hello World"), goczmq.FlagNone)
	if err != nil {
		panic(err)
	}

	frame, sz, err := pull.RecvFrame()
	if err != nil {
		panic(err)
	}

	fmt.Printf("We received a message of size %d\n", sz)
	fmt.Printf("The message was: '%s'\n", frame)

	pull.Destroy()
	push.Destroy()
}

Let’s go over what is happening in this short example in detail. First, we create a ZMQ_PUSH socket:

 	push, err := goczmq.NewPush("tcp://127.0.0.1:31337")
	if err != nil {
		panic(err)
	}

The NewPush function does quite a bit for us under the hood. Since it’s the first socket we’ve created in this program, it creates a ZeroMQ context for us. Then, it creates the socket and starts trying to connect the socket to the endpoint.

Note that we’ve constructed a TCP socket that is connecting before we’ve created the bound socket! With ZeroMQ, the order of bind and connect calls does not matter. The socket will retry connecting in the background until it is successful.

Next, we’ll construct a ZMQ_PULL socket which will bind to the endpoint. After it is bound, the ZMQ_PUSH socket will successfully connect to it.

	pull, err := goczmq.NewPull("tcp://*:31337")
	if err != nil {
		panic(err)
	}

Now comes the exciting part - sending a message! In ZMTP (the protocol implemented by ZeroMQ), a “message” consists of one or more byte agnostic frames. We will send a message by using the SendFrame API call. SendFrame accepts a []byte followed by flags. In this case, FlagNone indicates there are no frames following this frame, so the single frame should be treated as a full message. If this frame were the first frame in multi-part message, we would use goczmq.FlagMore.

	err = push.SendFrame([]byte("Hello World"), goczmq.FlagNone)
	if err != nil {
		panic(err)
	}

An important detail to note here is that ZeroMQ is asyncronous and provides a buffer under the hood - so our program execution continues after the call to SendFrame even though the message has not yet been received. This is what allows this example to send and receive in the same thread.

Now it’s time to receive the message:

	frame, sz, err := pull.RecvFrame()
	if err != nil {
		panic(err)
	}

	fmt.Printf("We received a message of size %d\n", sz)
	fmt.Printf("The message was: '%s'\n", frame)

After we’re done with the sockets, we call sock.Destroy to clean up memory. While Go is a garbage collected language, we’re wrapping a C API, so we need to clean up after ourselves:

	pull.Destroy()
	push.Destroy()

Next Steps

ZeroMQ is a large topic, and ZeroMQ combined with Go doubly so. I’ll be working on a series of articles covering the usage of interesting parts of the GoCZMQ API as well as lessons learned from working on the project.

Resources

20 Sep 2015, 07:54

Problem: I Want To Write To Redis From Rsyslog

Solution: Use the omhiredis output plugin.

I recently saw Radu Gheorghe express an interest in feeding Logstash from Rsyslog via a Redis queue. I realized that I’d neglected the omhiredis plugin since I’d original written it as a proof of concept. This seemed like a good opportunity to revisit the plugin and start fixing it up. For the rsyslog 8.13 release I added support for LPUSH and PUBLISH.

Building

Requirements

Configure rsyslog with omhiredis support:

~/git/rsyslog> ./configure --enable-omhiredis <other options>
~/git/rsyslog> make
~/git/rsyslog> sudo make install

Usage

Queue Mode

In queue mode, omhiredis will LPUSH each message into a Redis list stored at the defined key. You may define an optional template using the “template” parameter. If a template is not defined, the action will default to RSYSLOG_ForwardFormat.

module(load="omhiredis")

action(
  name="push_redis"
  type="omhiredis"
  mode="queue"
  key="testqueue"
)

Here is an example of RPOPing a log line from Redis that was inserted using the above action:

> redis-cli 
127.0.0.1:6379> RPOP my_queue

"<46>2015-09-17T10:54:50.080252-04:00 myhost rsyslogd: [origin software=\"rsyslogd\" swVersion=\"8.13.0.master\" x-pid=\"6452\" x-info=\"http://www.rsyslog.com\"] start"
127.0.0.1:6379> 

Publish Mode

In publish mode, omhiredis will PUBLISH each message onto a Redis channel stored at the defined key. Like queue mode, you may define a template using the “template” parameter, and if not set the template defaults to RSYSLOG_ForwardFormat.

module(load="omhiredis")

action(
  name="push_redis"
  type="omhiredis"
  mode="queue"
  key="testqueue"
)

Here is an example of subscribing to the channel defined in the above template, and receiving a published message from it:

> redis-cli 
127.0.0.1:6379> subscribe my_channel

Reading messages... (press Ctrl-C to quit)

1) "subscribe"

2) "my_channel"

3) (integer) 1

1) "message"

2) "my_channel"

3) "<46>2015-09-17T10:55:44.486416-04:00 myhost rsyslogd-pstats: {\"name\":\"imuxsock\",\"origin\":\"imuxsock\",\"submitted\":0,\"ratelimit.discarded\":0,\"ratelimit.numratelimiters\":0}"

Template Mode

In template mode, omhiredis will send the message constructed by the template directly to Redis as a command. Originally, this was the only mode the plugin supported. Please note that there is an outstanding bug in this mode - it will not properly handle commands with spaces in the message payload. For example, manually constructing an LPUSH of a full message using template mode will not work properly. I hope to find the time to fix this in the future. Pull requests are of course accepted!

Here’s an example of keeping a tally of the number of messages seen by program name. Note that mode is not set, as it’s the default mode for this module. Additionally, there’s a config parsing bug that is triggered if you explicitly set it, which will be fixed in 8.14.

module(load="omhiredis")

template(
  name="program_count_tmpl"
  type="string"
  string="HINCRBY progcount %programname% 1"
)

action(
  name="count_programs"
  server="my-redis-server.example.com"
  port="6379"
  type="omhiredis"
  template="program_count_tmpl"
)

Here, we take a look at the counts stored in Redis:

> redis-cli 
127.0.0.1:6379> HGETALL progcount
1) "rsyslogd"
2) "35"
3) "rsyslogd-pstats"
4) "4302"

Pipelining with queue.dequeuebatchsize

The omhiredis plugin supports pipelining using Rsyslog queuing with the queue.dequeuebatchsize. Note that the plugin does not currently check for errors in the replies. If the plugin becomes something people are seriously interested in, error handling should definitely be added. For those interested in internals, here’s the current end transaction block in the code:

BEGINendTransaction
CODESTARTendTransaction
    dbgprintf("omhiredis: endTransaction called\n");
    int i;
    pWrkrData->replies = malloc ( sizeof ( redisReply* ) * pWrkrData->count );
    for ( i = 0; i < pWrkrData->count; i++ ) {
        redisGetReply ( pWrkrData->conn, (void *)&pWrkrData->replies[i] );
        /*  TODO: add error checking here! */
        freeReplyObject ( pWrkrData->replies[i] );
    }
    free ( pWrkrData->replies );
ENDendTransaction

The rest of the source may be viewed on github.

Future Plans

I believe the next steps for this plugin should be:

  • Fixing “template” mode so that message payloads containing spaces work properly.
  • Error handling to prevent message loss on redis failure.
  • A complimentary “imhiredis” plugin that can receive from Redis

19 Sep 2015, 19:13

Problem: We (Software Developers) Attack The Work and Energy Of Others

Solution: Be aware of this pattern and don’t participate in it.

There is a destructive pattern that I see often among free software developers. It expresses itself through attacks on the work of others. As far as I can tell, the impedus behind it is a desire to secure status by expressing the idea that the work of others is contemptable. People will enter into this pattern together, reinforcing the idea that they are in the high status group while deriding the efforts of others.

In my own experience observing this pattern, the targets of these attacks are usually people involved with communities around technologies that are relatively welcoming to novices. The fact that these communities have produced impressive feats of software requiring deep expertise and domain knowledge is ignored in favor of belittling technology choices. Often, the flaws of the technology used will be exaggerated, giving the impression that it is “impossible” to create anything of value using it. The attitude projected is usually one of aloof, jaded cynicism. The posture is usually “playing to the crowd”.

Unfortunately, this behavior is especially damaging when aimed at novices, as they may lack the foundation of confidence necessary to keep from internalizing the idea that the knowledge they have aquired so far lacks value. These attacks plant seeds of doubt and undermine the learning process.

Maximizing the set of problems solved by software and finding the most accurate solution for each problem requires diverse participants with differing (and often conflicting) perspectives. Attacks on novices as well as attacks on novel approaches to software development help reinforce monocultures which severely limit the inputs into collaborative processes, which in turn severely limits the potential outputs. This pattern is anti-social, and puts the ego and status of individuals over the goals of collaboration, problem solving, and knowledge creation.

Since human beings engage in patterns of behavior because the behavior is rewarded, the solution to this problem is to create high status organizations with cultures that do not reward these behaviors. As individuals a simple step we can take is to not participate in nor reward others who participate in this behavior, and to point it out for what it is when we see it.

There are too many problems that need to be solved to waste time and energy deliberately doing damage to individuals and communities who are actively engaged in problem solving.

19 Sep 2015, 17:31

Problem: Starting Each Post With 'Problem' May Seem Odd

Solution: Explain why we’re doing this.

One of my favorite free software communities that I participate in is the ZeroMQ project. This is a distributed organization of programmers working on a software ecosystem around ZeroMQ, and using it to solve different problems. We develop software within this organization using a process called the Collective Code Construction Contract. This is a software development process designed with the goal of maximizing the scale of community around a project. If you’re curious as to how well it’s working, I suggest reading “C4.1 - an Exothermic Process” by the ZeroMQ organization’s benevolent dictator, Pieter Hintjens.

One of the core practices we use within this methodology is limiting patches to code to the scope of solving a narrowly defined problem. Solving more than one problem with a patch is cheating. If a patch identifies a problem, solves it, and meets some basic style guidelines, it gets accepted. Additionally, the author of the patch is made a new maintainer within the organization. We do not believe in high barriers to entry to our organization, as it would be counter to the goal of maximizing the scale of the community.

I have found getting into the mindset of identifying individual problems and solving them minimally and accurately to be a wonderful tool for maintaining focus. So, I thought it would be fun to try this technique on my blog as well. If I’m expending the effort to write, there most be a reason I’m doing so. So, even if the problem I’m addressing is “I want to look smart about a subject”, or “my ego likes it when people read things I write”, I’m still going to identify the problem each post is addressing. I think it will be fun.

If you’d like to learn more about the C4 process (it’s actually the C4.1 process now, as we patched it) I highly recommend watching the talk “Building consistently good software with ordinary people”. It’s one of my favorite talks about collaboration.

19 Sep 2015, 11:16

Problem: There Are No Posts

Solution: Write a ‘Hello World’ post.

Greetings, and welcome to the Oldschool System blog. Thank you so much for taking the time to stop by here and read my thoughts. I’m flattered, and will endeavor to write things that are worth your time.

The sorts of things you are most likely to find here are:

  • Information about the free software projects I participate in
  • Examples of how to do things I find fun with programming languages
  • Thoughts on the culture(s) that have formed around software engineering
  • Political musings, mostly around the intersection of my craft and politics

If these are things that might interest you for some reason, check in now and then. If these are things that don’t seem very interesting, no offense taken, and enjoy the rest of your journey.

19 Sep 2015, 10:45

Problem: People don't know who I am

Solution: Make an ‘About’ page.

A Quote I Like

“FYI man, alright. You could sit at home, and do like absolutely nothing, and your name goes through like 17 computers a day. 1984? Yeah right, man. That’s a typo. Orwell is here now. He’s livin’ large. We have no names, man. No names. We are nameless!” - Cereal Killer (from the movie “Hackers”)

Projects

Find Me

me