Sample data patterns

This topic provides sample data patterns that you can help you better understand the process of data-pattern creation. Study these patterns before you create new data patterns or customize existing ones.

Each sample contains sample data from a log file along with the user input (date format and primary pattern) that you will need to provide when creating a new data pattern. Also, you can correlate the sample data and the primary pattern to understand the fields extracted and available for search.

Note

At the time of indexing, the details field is ignored. It is used to assign miscellaneous information in your data that you do not want to categorize with specific fields. All name=value pairs in the data to which this field is applied are extracted as fields.

This topic contains the following sample data patterns:

Data pattern sample 1

Pattern name	Log4J
Date format	EEE MMM dd HH:mm:ss Z yyyy
Primary pattern	%{Log4JTimestamp:timestamp}\s+:?\s+%{MultilineEntry:details}
Sample data	Thu Aug 09 10:18:42 Eastern Daylight Time 2012 : Rendering view [org.springframework.web.servlet.view.RedirectView: unnamed; URL [/pets/1]] in DispatcherServlet with name 'petclinic' Thu Aug 09 10:19:52 Eastern Daylight Time 2012 : Successfully completed request
Fields extracted	From line 1: timestamp = Thu, 09 Aug 2012 14:18:42 GMT details = Rendering view [org.springframework.web.servlet.view. RedirectView: unnamed; URL [/pets/1]] in DispatcherServlet with name 'petclinic' From line 2: timestamp = Thu, 09 Aug 2012 14:19:52 GMT details = Successfully completed request

Data pattern sample 2

Pattern name	IBM WebSphere - SystemError
Date format	MM/dd/yy HH:mm:ss:SSS Z
Primary pattern	\[%{IbmWebsphereTimestamp:timestamp}\] \s%{Data:groupid}\sSystemErr\s+%{Data:level} \s+(?:at\s+%{GreedyData:class} \.%{Data:function}\((?:.:%{Data:linenum}\|.)\) \|%{MultilineEntry:details})
Sample data	[5/4/12 16:14:07:113 PDT] 00000025 SystemErr R com.ibm.ws.exception.RuntimeError: java.lang.RuntimeException: java.lang.NoClassDefFoundError: com.ibm.lang.management.MemoryMXBeanImpl (initialization failure) [5/4/12 16:14:07:113 PDT] 00000025 SystemErr R at com.ibm.ws.runtime.component. ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:789)
Fields extracted	From line 1: timestamp = Fri, 04 May 2012 23:14:07 GMT groupid = 00000025 level = R details = com.ibm.ws.exception.RuntimeError: java.lang.RuntimeException: java.lang.NoClassDefFoundError: com.ibm.lang.management.MemoryMXBeanImpl (initialization failure) From line 2: timestamp = Fri, 04 May 2012 23:14:07 GMT groupid = 00000025 level = R class = com.ibm.ws.runtime.component.ApplicationMgrImpl function = startApplication linenum = 789

Data pattern sample 3

Pattern name	MySQL - Error
Date format	yyMMdd HH:mm:ss
Primary pattern	%{MysqlErrorTimestamp:timestamp}\s+ %{Data:message}\sVersion:%{Data:version}\s+socket:\s %{Data:socket}\s+port:\s*%{Port:portnumber}\s %{MultilineEntry:details}
Sample data	070102 16:19:29 InnoDB: Started; log sequence number 0 43644 /usr/libexec/mysqld: ready for connections. Version: '4.1.10a-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution 070102 16:20:29 InnoDB: Started; log sequence number 0 43644 /usr/libexec/mysqld: ready for connections. Version: '4.1.10a-log' socket: '/var/lib/mysql/mysql.sock' port: 3307 Source distribution
Fields extracted	From line 1: timestamp = Tue, 02 Jan 2007 10:49:29 GMT message = InnoDB: Started; log sequence number 0 43644 /usr/libexec/mysqld: ready for connections. version = '4.1.10a-log' socket = '/var/lib/mysql/mysql.sock' portnumber = 3306 details = Source distribution From line 2: timestamp = Tue, 02 Jan 2007 10:50:29 GMT message = InnoDB: Started; log sequence number 0 43644 /usr/libexec/mysqld: ready for connections. version = '4.1.10a-log' socket = '/var/lib/mysql/mysql.sock' portnumber = 3307 details = Source distribution

Data pattern sample 4

Pattern name	ITDA
Date format	MMM dd, yyyy hh:mm:ss a
Primary pattern	%{ITDATimestamp:timestamp}\s+%{Data:class}\s+ %{Data:function}\(\):%{Int:linenum}\s+\n* (?:%{ITDADebugLevel:level}:\s*%{MultilineEntry:details})?
Sample data	May 28, 2014 02:14:52 PM org.elasticsearch.common.logging.slf4j. Slf4jESLogger internalWarn():110 WARN: [Mangle] master_left and no other node elected to become master, current nodes: {[Mangle][gUBYCkO8RBaiZ2r6seK_UQ][PTL2662] [inet[/10.88.196.37:9306]]{client=true, data=false},} May 28, 2014 02:14:55 PM com.bmc.ola.webclient.CompleteRequestExecution getReadyReponses():87 ERROR: execution of request interrupted
Fields extracted	From line 1: timestamp = Wed, 28 May 2014 08:44:52 GMT class = org.elasticsearch.common.logging.slf4j.Slf4jESLogger function = internalWarn linenum = 110 level = WARN details = [Mangle] master_left and no other node elected to become master, current nodes: {[Mangle] [gUBYCkO8RBaiZ2r6seK_UQ][PTL2662] [inet[/10.88.196.37:9306]]{client=true, data=false},} client = true data = false From line 2: timestamp = Wed, 28 May 2014 08:44:55 GMT class = com.bmc.ola.webclient.CompleteRequestExecution function = getReadyReponses linenum = 87 level = ERROR details = execution of request interrupted

Data pattern sample 5

Pattern name	Cisco Syslog
Date format	MMM dd yyyy HH:mm:ss
Primary pattern	%{CiscoTimestamp:timestamp}:\s\%%{TGenerator:generator}- %{PosInt:level}-%{PosInt:messagenumber}:\s* (?:\|%{MultilineEntry:details})
Sample data	Jul 14 2013 09:54:18: %PIX-6-302005: Built UDP connection for faddr 198.207.223.240/53337 gaddr 10.0.0.187/53 laddr 192.168.0.2/53 Jul 14 2013 09:54:26: %PIX-4-106023: Deny icmp src outside: Some-Cisco dst inside:10.0.0.187 (type 3, code 1) by access-group "outside_access_in"
Fields extracted	From line 1: timestamp = Sun, 14 Jul 2013 04:24:18 GMT generator = PIX level = 6 messagenumber = 302005 details = Built UDP connection for faddr 198.207.223.240/53337 gaddr 10.0.0.187/53 laddr 192.168.0.2/53 From line 2: timestamp = Sun, 14 Jul 2013 04:24:26 GMT generator = PIX level = 4 messagenumber = 106023 details = Deny icmp src outside:Some-Cisco dst inside:10.0.0.187 (type 3, code 1) by access-group "outside_access_in"

Data pattern sample 6

Pattern name	Access Log - Combined
Date format	dd/MMM/yyyy:HH:mm:ss z
Primary pattern	%{Data:info}\s%{IpOrHost:ip}\s%{Data:rfc931}\s %{Data:username}\s\[%{AccessCombinedTimestamp:timestamp}\]\s %{Data:request}\s%{PosInt:statuscode}\s%{PosInt:bytes}\s %{Data:referrer}\s%{AnyStringInQuotes:useragent}\s %{Data:cookie}(?:\|%{MultilineEntry:details})
Sample data	"66.249.66.102.1124471045570513" 59.92.110.121 - - [15/Jul/2013:10:04:01 -0700] "GET /themes/images/apache_pb.gif HTTP/1.1" 200 994 "http://www.example.com/index.html" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050524 Fedora/1.0.4-4 Firefox/1.0.4" "61.3.110.148.1124404439914689" "66.249.66.102.1124471045570513" 59.92.110.122 - - [15/Jul/2013:10:04:02 -0700] "GET /themes/images//apache_bg.gif HTTP/1.1" 200 2323 "http://www.example.com/index.html" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050524 Fedora/1.0.4-4 Firefox/1.0.4" "61.3.110.148.1124404439914689"
Fields extracted	From line 1: info = "66.249.66.102.1124471045570513" ip = 59.92.110.121 rfc931 = - username = - timestamp = Mon, 15 Jul 2013 17:04:01 GMT request = "GET /themes/images/apache_pb.gif HTTP/1.1" statuscode = 200 bytes = 994 referrer = "http://www.example.com/index.html" useragent = "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050524 Fedora/1.0.4-4 Firefox/1.0.4" cookie = details = "61.3.110.148.1124404439914689" From line 2: info = "66.249.66.102.1124471045570513" ip = 59.92.110.122 rfc931 = - username = - timestamp = Mon, 15 Jul 2013 17:04:02 GMT request = "GET /themes/images//apache_bg.gif HTTP/1.1" statuscode = 200 bytes = 2323 referrer = "http://www.example.com/index.html" useragent = "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050524 Fedora/1.0.4-4 Firefox/1.0.4" cookie = details1 = "61.3.110.148.1124404439914689"