Log Parser


 

These tools were created based on personal experience and suggestions from co-workers and friends over the years.  They are provided for free and can be distributed at will, for private or commercial use.

Disclaimer of Warranty,

THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

Limitation of Liability,

IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

All of these programs have Batch files that allow for drag and drop of your log files onto the batch file to produce the output.  The Java command lines are for anyone wanting to run the commands yourself, but shouldn’t be necessary if running windows.

 

Parse Server Side Log

This utility was written to allow a single thread to be pulled from a multi-thread log file.  This program has one required command line parameter, and one optional.  The first parameter is the name of the log file to be processed. The second is optional, and is the thread ID to be pulled out of the log file.  If the second parameter is not provided, then it must be provided in the ParseServerSideLog.cfg file.  An example of calling this program from command line would be

java -cp .\lib\parse_logs.jar com.apl.ParseServerSideLog arfilter.log 0000008036 > output.log

This command line assumes that the jar file is located in a lib folder under the current folder, that the log file to be parsed is local in the current directory, and that thread 8036 exists as a thread in the input log.  It will output the values that match that thread into a redirected file of output.log.

If you want to utilize the program with the provided cfg file, the cfg file would be updated to contain

threadID:             0000008036

Then, you would be able to provide the following command line

java -cp .\lib\parse_logs.jar com.apl.ParseServerSideLog arfilter.log > output.log

This provides the same output as above, but allows for use of the ‘drag and drop’ functionality of log file onto batch file (provided in zip)

 

Parse String

     This utility was written as a derivative of Parse Server Side Log.  But instead of being hard coded to just pull out a Thread ID, it is designed to be able to pull out any string you want to pull out of a text file.  Additionally, it has another parameter that is used that allows the string in question be removed from the resulting file instead of be the only content of the file.  This program requires the use of a command line parameter, and a config file named ParseString.cfg.  An example of calling this program from command line would be

java -cp .\lib\parse_logs.jar com.apl.ParseString arfilter.log > output.log

This command line assumes that the jar file is located in a lib folder under the current folder, that the log file to be parsed is local in the current directory.  The config file has two parameters:

string:   <string you want to look for>

You simply set the string to the value you want to look for

makeNegative: true/false

If set to true, the system will exclude any line that contains your string, if false, only your string will be returned

The batch file provided in the zip allows for  ‘drag and drop’ functionality of log file onto batch file.

Remove Failed/Disabled

This utility came about because of a discussion with a co-worker about how much easier it would be to read workflow logs if we didn’t need to look at all of the workflow that didn’t actually do anything, so this would be workflow that had a run-if that failed qualification, and there were no else actions, or things that were skipped because of GoTo actions, basically, making the file as clean as possible to only look at things that WERE happening.  This program has one command line parameter, which is the file you want to process

java -cp .\lib\parse_logs.jar com.apl.RemoveFailedDisabled arfilter.log > output.log

This command line assumes that the jar file is located in a lib folder under the current folder, that the log file to be parsed is local in the current directory.

The resultant output log file should be more concise than the file you started with

 

Remove Thread Info

This program came out of frustration of constantly needing to scroll left and right in my text editor to be able to see what workflow was doing, especially in the case of Active Links and Filters being logged in the same file.  This program leaves the <Function> tags at the beginning of each line, but removes everything between that and the actual data in the log, removing all of the thread and related information.  This program has one command line parameter, which is the file you want to process

java -cp .\lib\parse_logs.jar com.apl.RemoveThreadInfo arfilter.log > output.log

This command line assumes that the jar file is located in a lib folder under the current folder, that the log file to be parsed is local in the current directory.

The resultant output log file should be more ‘horizontally compact’ than the original file.  You may want to run this utility AFTER you have isolated the transaction in question, because the thread, rpc queue, user information will be removed after this utility is run

 

Split Server Side Log

This utility came because of a need to not parse out a single thread from a server side log, but to be able to take all threads and put them into separate files.  We all know that this can be done at the time of logging by turning on the ‘Log Per Thread’ option on the logging tab, but I have found that option to be ‘ugly’ for most logging, and have historically not used the option much, but this program gives you the ability to take a single log file produced without that option turned on, and generate the same files it would have if the option was turned on at capture time.  This program has one command line parameter, which is the file you want to process

java -cp .\lib\parse_logs.jar com.apl.SplitServerSideLog arfilter.log

This command line assumes that the jar file is located in a lib folder under the current folder, that the log file to be parsed is local in the current directory.

This program is a bit different than the others because you aren’t providing an output file redirect. This program creates one file for each thread in the source file, and names the file with the thread id.log as the name.  The output from this program on the display is relatively simple.  For every 1000 lines it processes, it puts a . on the screen, and it puts 100 dots per line, making each line worth 100,000 log lines that it parsed.  It keeps track of the number of lines that it has output to the screen so you can keep rough count of how many lines it has processed.

 

Transform RegEx

This program began as a derivative of the Remove Thread Info program.  What this program does is it takes a ‘Regular Expression’, or RegEx, and performs a translation on the source row converting what you search for into what you want it to be.  This program has one command line parameter, which is the file you want to process

java -cp .\lib\parse_logs.jar com.apl.TransformRegEx arfilter.log > output.log

This command line assumes that the jar file is located in a lib folder under the current folder, that the log file to be parsed is local in the current directory.

This program requires the TransformRegEx.cfg to be in the same location as the program.  This config file has two parameters:

regEx: > <TID.* > /\\*

regEx is the Java Regular Expression that you want to search a line for, the example above is looking for the string “ > <TID”, then the .* indicates to look as far as it needs with as many characters as possible (the last instance of) “ > /*”….but because this is programming, and * is a ‘special’ character to Regular Expressions, the * needs to be turned into \* (escaping the character)….then to double whammy things, \ is also a special character, so in this case, when looking for “ > /”, we need to store “ > /\\*”, which properly translates at the Java level from config file to program.

replace: > /\\*

replace is just string we want it to replace the string it found with, remembering that we need to escape ‘special’ RegEx characters. The above regEx and replace transform

<FLTR> <TID: 0000035508> <RPC ID: 0057089889> <Queue: Admin     > <Client-RPC: 390680   > <USER: Remedy Application Service > /* Tue Jul 30 2013 08:18:04.1380

Into

<FLTR> /* Tue Jul 30 2013 08:18:04.1380

 

This utility suite was presented @ WWRUG13.  I hope you all find it as useful as I have over the years.

 

Version History

1.1 – October 30 2013

-Removed trim from ‘RemoveThreadInfo’ because it was messing up 8.1 Web Logging indentation

-Updated ‘RemoveFailedDisabled’ to handle Failed/Disabled Active Links in 8.1 Web Logging

1.0 – October 1 2013

Initial Release of 6 tools

-ParseServerSideLog

-ParseString

-RemoveFailedDisabled

-RemoveThreadInfo

-SplitServerSideLog

-TransformRegEx