While I’m playing with Javascript and React, I’m also going on with using Python and Flask to build a backend. To have meaningful statistics about my app usage and what to optimize, I wanted to centralize logging somewhere.

Context

When I read Python Microservices Development, one of the options for logging was Graylog, but the description to use it was for Graylog 2, and Graylog 3 was just out.

Furthermore, the documentation for Docker was explaining how to build with an older version of Docker Compose that didn’t support link, but it’s not working anymore!

So it makes sense to have a full tutorial, from MongoDB set up to ElasticSearch.

MongoDB setup

Let’s start with MongoDB set up. It’s very easy as long as we add an init.sh script that will create the database for Graylog:

mongo admin -u $MONGO_INITDB_ROOT_USERNAME -p $MONGO_INITDB_ROOT_PASSWORD --eval "db.getSiblingDB('graylog').createUser({user: 'graylog', pwd: 'xxxxx', roles: [{role: 'readWrite', db: 'graylog'}]});"

We can now modify out Docker image to launch the script at start up:

ADD init.sh /docker-entrypoint-initdb.d/init.sh
ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
EXPOSE 27017
CMD ["mongod"]

This will properly set up the MongoDB the first time the image is run.

Graylog setup

Let’s now move to Graylog set up through Docker Compose.

We start by generating another password and its SHA hash:

pwgen -N 1 -s 96
echo -n yourpassword | shasum -a 256

We fill this inside the Docker Compose configuration:

  graylog:
    image: graylog/graylog:3.0
    environment:
      GRAYLOG_PASSWORD_SECRET: xxxx
      GRAYLOG_ROOT_PASSWORD_SHA2: xxxx
      GRAYLOG_HTTP_EXTERNAL_URI: http://192.168.0.1:9000/
      GRAYLOG_MONGODB_URI: mongodb://graylog:xxxx@nosql:27017/graylog
      GRAYLOG_ELASTICSEARCH_HOSTS: http://elasticsearch:9200
    depends_on:
      - nosql
      - elasticsearch
    ports:
      - 9000:9000
      - 12201: 12201

Set GRAYLOG_HTTP_EXTERNAL_URI to the address from which you will serve the Graylog service. Otherwise, you will get a blank page!

We add the rest of the configuration as well:

  nosql:
    image: my-nosql
    volumes:
      - local_storage_nosql:/data/db
    environment:
    # provide your credentials here
    - MONGO_INITDB_ROOT_USERNAME=root
    - MONGO_INITDB_ROOT_PASSWORD=xxx
    
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.6.2
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"

Launching and using Graylog

After launching Graylog, you can go at http://192.168.0.1:9000 to start using Graylog. It is possible to use graypy or pygelf, I’m going to show how to use the first former, but the latter works just as well with Graylog 3.

Let’s create a small Python script to log something:

import logging
import graypy

my_logger = logging.getLogger('test_logger')
my_logger.setLevel(logging.DEBUG)

handler = graypy.GELFUDPHandler('192.168.0.1', 12201)
my_logger.addHandler(handler)

my_logger.debug('Hello Graylog2.')

But if you look at Graylog webpage, you won’t see anything. The reason is that you need to create an input first.

In the admin panel, go to System->Inputs and see that you don’t have any existing input. To create one, select GELF UDP in the drop-down and click on the now available Launch new input. Check Global to start the input on all nodes and launch your script again.

You should now see messages coming in.

Conclusion

The configuration is quite straightforward, but if you don’t have all the pieces, then you won’t see the results or the logs coming in. Now you can start logging and explore Graylog further!