Logs are very important for any application for debugging. Logs are the first and many times last tool available to get it to the root cause of any issue.
In this blog I am not going to discuss about what to log and how to log effectively. But I will walk through the critical aspect of logging which is log levels. In every environment whether it is production/QA/Dev each one has particular log levels configured. In production environment log levels are strictly managed to make sure log file size is minimum.
Every typical day is not same, bugs are unavoidable in software development. Very often a situation occurs where the current logs which are getting logged at particular level are insufficient. So, need arise to bump up the log levels.
In poorly designed application where log files are inbuild (either in the jar or image) it altogether needs new build to merely change the log levels. In decently design application log configuration are externalized. But still to merely change the log levels it need application restart.
In terms of scaling challenge is to change log levels in each instance. So to avoid all pitfalls following is needed:
- Dynamically change the log level at runtime without application restart.
- Propagation of log level changes across the application instances.
Logging frameworks has undergone lots of changes in the past 5 years and we are going to leverage that. As every one moving towards microservice architecture we are inevitably going to adhere that and will use Spring Boot, sl4j and Logback as our application stack.
Following are the options to achieve that. Usage is not only restricted to microservice architecture it can be applied in other style of architecture too provided we are using the sl4j and Logback.
Option -1 Using the Spring Boot Admin
This is the best and most recommended option if you are using Spring Boot as everything comes out of the box. In this option your application needs to enable spring actuator and get it registered with Spring Boot Admin.
User will use the spring boot admin UI to change the log levels.
Merely select the logger and change the log level in just click on a button. As highlighted in the figure it will change log level in all running instances you no need to perform anything in addition.
If you are new to spring boot admin you can get details here. You can find entire working code here.
Option -2 Scan the logback.xml
You can configure Logback to periodic scan and update the logging configuration via logback.xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" scanPeriod="15 seconds">
...
</configuration>
If you alter the log levels in logback.xml file then changes will get effect max after 15 seconds. To propagate the log level changes across all container instance at once follow the below architecture:
Over here you externalize the logback.xml file and make it available to the container via NFS mount point. Pass the log file path to the container via JVM argument: ” -Dlogging.config=/logpath/logback.xml”.
Option -3 Access logback.xml via URL
In this option you need exposing the logback.xml via URL with the help of nginx/apache/HAProxy etc. and application will consume the logback.xml via JVM arg: “-Dlogging.config = http://host:9999/logback.xml”. Woefully, auto scan doesn’t work if you expose the logback.xml via URL. So to achieve that we need some polling mechanism.
public void load() throws JoranException, MalformedURLException, InterruptedException {
String prev = null;
URL url = new URL("http://localhost:9999/logconfig/logback.xml");
while (true){
if(!(getNewContentIfChanged(prev,url) == null)){
LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();
context.reset();
JoranConfigurator configurator = new JoranConfigurator();
configurator.setContext(context);
configurator.doConfigure(url);
System.out.println("Log reloaded >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>");
}
TimeUnit.SECONDS.sleep(15);
}
}
Log level changes propagation:
Option-4 Creating API to change the log levels
Another way is to change the log level programmatically by exposing the API.
@RequestMapping("/changelevel")
public String changeLogLevel(@RequestParam String loggerName, @RequestParam String level){
LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory();
ch.qos.logback.classic.Logger logger = loggerName.equalsIgnoreCase("root")?
loggerContext.getLogger(loggerName):loggerContext.exists(loggerName);
if( logger !=null){
logger.setLevel(Level.toLevel(level));
return "Changed logger: "+loggerName+" to level : "+level;
} else {
return "Logger Not Found Make Sure that logger name is correct";
}
}
To propagate the log level changes across the instances we need to follow one of the methods below:
We need custom utility which will expose the API to change the log level. Utility will fetch the instance/s IP address and execute the API call on each running APP instances.
Option – 5 Pub / Sub
In this option the application instance needs to subscribe to the topic where log level changes will get published.
As per me this is termed over engineering if you are doing for just log level changes. But this can be very effective in microservice architecture if you want to change the log levels of all applications in one shot.
Conclusion
There are other ways to expose and load the log configuration via ftp, socket, etc. but the approach is the same. Which approach to take it is utterly dependent on the deployment topology, application architecture style and need.
Above represent the approaches but securing the log configuration and auditing the changes are out of the scope of the blog. I am assuming you will take care in your respective environment.
Hope you enjoy the post. Feel free to click like button.
Entire code you can find here.
Happy coding.
Coding Addicted 