Kafka logging for Gluu

Overview

This tutorial offers a step-by-step guide for using Apache Kafka to aggregate logs for oxAuth and oxTrust.

Process

1. From inside the chroot, unzip the oxAuth and Identity war files.

   # unzip -p /opt/gluu/jetty/oxauth/webapps/oxauth.war WEB-INF/classes/log4j2.xml > /opt/gluu/jetty/oxauth/resources/log4j2.xml
   # unzip -p /opt/gluu/jetty/identity/webapps/identity.war WEB-INF/classes/log4j2.xml > /opt/gluu/jetty/identity/resources/log4j2.xml
   

2. Download the Kafka Clients jar file and copy it into oxAuth and Identity.

   # wget http://central.maven.org/maven2/org/apache/kafka/kafka-clients/2.1.1/kafka-clients-2.1.1.jar -O /tmp/kafka-clients-2.1.1.jar
   # cp /tmp/kafka-clients-2.1.1.jar /opt/gluu/jetty/oxauth/custom/libs/
   # cp /tmp/kafka-clients-2.1.1.jar /opt/gluu/jetty/identity/custom/libs/
   

3. Make the following changes to log4j2.xml in both /opt/gluu/jetty/oxauth/resources and /opt/gluu/jetty/identity/resources:

   • Under section appenders, add:

     <Kafka name="Kafka" topic="my-gluu-topic">
     <PatternLayout pattern="%d %-5p [oxauth] [%t] [%C{6}] (%F:%L) - %m%n"/>  
     <Property name="bootstrap.servers"=>KAFKA-1.MY.DOMAIN:9094,KAFKA-2.MY.DOMAIN:9094,...</Property>
     
     <Property name="compression.type">gzip</Property>
     <Property name="ssl.truststore.location">/PATH/TO/truststore.jks</Property>
     <Property name="ssl.truststore.password">PASSWORD_FOR_TRUSTSTORE</Property>
     <Property name="security.protocol">SSL</Property>
     </Kafka>
     

   • Under each logger, including the root logger, in sections that have an AppenderRef, add:

     <AppenderRef ref="Kafka" />
     

4. Add "-dlog4j.configurationfile=resources/log4j2.xml" to java_options in both /etc/default/oxauth and /etc/default/identity

5. Restart systemd, oxAuth, and Identity

   # service oxauth restart && service identity restart
   # logout
   # systemctl daemon-reload