This guide walks you through the steps for creating a new Elastic Beat. The Beats are a collection of lightweight daemons that collect operational data from your servers and ship it to Elasticsearch or Logstash. The common parts for all Beats are placed in the libbeat library, which contains packages for sending data to Elasticsearch and Logstash, for configuration file handling, for signal handling, for logging, and more. By using this common framework, you can ensure that all Beats behave consistently and that they are easy to package and run with common tools.
Get Ready
All Beats are written in Go, so having Go installed and knowing the basics are prerequisites for understanding this guide. But don’t worry if you aren’t a Go expert. Go is a relatively new language, and very few people are experts in it. In fact, several people learned Go by contributing to Packetbeat and libbeat, including the original Packetbeat authors.
After you have installed Go and set up the GOPATH environment variable to point to your preferred workspace location, a simple way of getting the source code for Topbeat and libbeat and compiling them at the same time is to do:
go get github.com/elastic/Topbeat
In this guide, we use working examples from the Topbeat source code to demonstrate how to implement a Beat. Topbeat is similar to the top command line tool, but instead of printing the statistics to the screen, Topbeat periodically ships them to Elasticsearch for storage.
You can use Topbeat as an example implementation for creating a new Beat. Just copy the source files and modify them as necessary for your Beat.
Overview
At the high level, a simple Beat like Topbeat has two main components:
- a component that collects the actual data, and
- a publisher that sends the data to the specified output, such as Elasticsearch or Logstash.
The publisher is already implemented in libbeat, so you typically only have to worry about the logic specific to your Beat (the code that creates the event and sends it to the publisher). Libbeat also offers common services like configuration management, logging, daemonizing, and Windows service handling, and in the future, will offer data processing modules, such as filtering or sampling.
The event that you create is a JSON-like object (Golang type map[string]interface{}) that contains the collected data to send to the publisher. At a minimum, the event object must contain a @timestamp field and a type field. Beyond that, events can contain any additional fields, and they can be created as often as necessary.
The following example shows an event object in Topbeat:
{
count":1,
"proc.cpu":{
"user":20,
"percent":0.983284169124877,
"system":0,
"total":20,
"start_time":"20:20"
},
"proc.mem":{
"size":333772,
"rss":6316,
"share":2996
},
"proc.name":"topbeat",
"proc.pid":13954,
"proc.ppid":10027,
"proc.state":"sleeping",
"shipper":"vagrant-ubuntu-trusty-64",
"@timestamp":"2015-08-06T20:20:34.089Z",
"type":"proc"
}
Now that you have the big picture, let’s dig into the code.
Developing the Beat-Specific Code
The Beat-specific code should implement the Beater interface defined in libbeat.
type Beater interface {
Config(*Beat) error
Setup(*Beat) error
Run(*Beat) error
Cleanup(*Beat) error
Stop()
}
This means your Beat should implement the following methods:
Config
Deals with the configuration file and optionally with custom CLI flags
Setup
Contains logic that executes before the main loop, usually for initialization
Run
Contains the main application loop that captures data and sends it to the publisher
Cleanup
Contains logic that executes after the main loop finishes (or is interrupted)
Stop
Contains logic that is called when the Beat is signaled to stop
The Beat parameter received by most of these methods contains data about the Beat, such as the name, version, and common configuration options.
Note: To be consistent with other Beats, you should append beat to your Beat name.
Let’s go through each of the methods in the Beater interface and look at a sample implementation.
Config Method
The Config method deals with the configuration file and optionally with custom CLI flags.
The recommended way of handling the configuration is to create a ConfigSettings type that matches the structure of the expected configuration file. Here is an example configuration section for Topbeat:
input:
period: 1
procs: [".*"]
And here are the corresponding Go structures, which are defined in config.go:
type TopConfig struct {
Period *int64
Procs *[]string
}
type ConfigSettings struct {
Input TopConfig
}
Pointers are used to distinguish between when the setting is completely missing from the configuration file and when it has a value that matches the type’s default value.
With these structures defined, the Config method looks like this:
func (tb *Topbeat) Config(b *beat.Beat) error {
err := cfgfile.Read(&tb.TbConfig, "")
if err != nil {
logp.Err("Error reading configuration file: %v", err)
return err
}
if tb.TbConfig.Input.Period != nil {
tb.period = time.Duration(*tb.TbConfig.Input.Period) * time.Second
} else {
tb.period = 1 * time.Second
}
[...]
return nil
}
Setup method
The Setup method enables you to execute logic before the main loop, usually for initialization. In the Topbeat implementation, this method only assigns the Beat object to the Topbeat object, so it doesn’t have to be passed to all sub-functions.
func (tb *Topbeat) Setup(b *beat.Beat) error {
tb.Beat = b
return nil
}
Run Method
The Run method should contain your main application loop. For Topbeat it looks like this:
func (t *Topbeat) Run(b *beat.Beat) error {
t.isAlive = true
t.initProcStats()
var err error
for t.isAlive {
time.Sleep(t.period)
err = t.exportSystemStats()
if err != nil {
logp.Err("Error reading system stats: %v", err)
}
[...]
}
return err
}
Inside the loop, Topbeat sleeps for a configurable period of time and then captures the required data and sends it to the publisher via the events publisher client. The publisher client is available as part of the Beat object through the Beat.Events variable.
The actual sending is done inside the exportSystemStats() method:
func (t *Topbeat) exportSystemStats() error {
load_stat, err := GetSystemLoad()
if err != nil {
logp.Warn("Getting load statistics: %v", err)
return err
}
[...]
event := common.MapStr{
"@timestamp": common.Time(time.Now()),
"type": "system",
"load": load_stat,
"cpu": cpu_stat,
"mem": mem_stat,
"swap": swap_stat,
}
t.Beat.Events.PublishEvent(event)
return nil
}
Cleanup Method
The Cleanup method is executed after the main loop finishes (or is interrupted) and gives you the opportunity to release any resources you might use. For Topbeat, the method is completely empty:
func (tb *Topbeat) Cleanup(b *beat.Beat) error {
return nil
}
Stop Method
The Stop method is called when the Beat is signaled to stop, for example through the SIGTERM signal on Unix systems or the service control interface on Windows. For Topbeat, this method simply sets isAlive to false, which breaks the main loop.
func (t *Topbeat) Stop() {
t.isAlive = false
}
The main Function
If you follow the Topbeat model and put your Beat-specific code in its own type that implements the Beater interface, the code from your main package is very simple:
func main() {
tb := &Topbeat{}
b := beat.NewBeat(Name, Version, tb)
b.CommandLineSetup()
b.LoadConfig()
tb.Config(b)
b.Run()
}