Using NXLog to Collect Windows Event Logs

There are a lot of syslog collectors for Windows, but when it comes to stability and features, NXlog has the best chances to fulfill all the requirements.

Windows EventLog allows multi-line messages, so this text is a lot more readable and nicely formatted by spaces, tabs and line-breaks as can be seen in Event Viewer. Because syslog only reads/writes single-line messages, this formatting must be stripped of the EventLog message. In doing so, we lose the meta-data. NXlog is capable of reading these fields, recognize the structure and forward these remotely (or act on them for alerting purposes), thus sparing you time and resources. So, if you use the NXlog framework (client/server) there will be no need to spend time writing patterns to extract usernames, IP addresses and similar meta-data.

Forwarding Windows Event Logs to Syslog Server (Kiwi Syslog)

Centralizing your logs saves time and increases the reliability of your log data, especially for Windows machines. When Windows log files are stored locally on each server, you have to individually log into each one to go through them and look for any errors or warnings. It’s possible for a Windows server to forward its events to a “subscribing” server. In this scenario the collector server can become a central repository for Windows logs from other servers in the network.There are many ways you can forward your windows event logs to a centralized log server. You can use event log forwarding feature which was introduced in Windows Server 2008. Event log forwarding brought forth a native and automatic way to get events from multiple computers (event sources) into one or more machines called collectors. Another option is to use third party software, such Solarwinds Free Event Log Forwarder for Windows. 

In this post, I am going to introduce another free software , Eventlog to Syslog. The Eventlog to Syslog utility is a program that runs on Microsoft Windows NT class operating systems monitoring the eventlog for new messages. When a new message appears in the eventlog, it is read, formatted, and forwarded to a UNIX syslog server.

1. Install Syslog Server - Kiwi Syslog Free Version
1.1 Download the Kiwi Syslog Daemon from the Kiwi from Download address: https://thwack.solarwinds.com/community/free-tools-and-trials
1.2 Run the Kiwi Syslog Daemon executable file to launch the installation program. Follow the instructions in the installation wizard to install the Kiwi Syslog Daemon as a service.
1.3 Once the Kiwi Syslog Daemon is installed, start the program to start the Syslog Daemon.

Gartner Magic Quadrant for SIEM Products (2018,2017,2016,2015,2014,2013,2012,2011...)

Gartner defines SIEM as a technology that aggregates data produced by security devices, network infrastructure and systems, and applications. Products in the security information and event management (SIEM) market analyze security event data and network flow data in real time for internal and external threat management. They collect, store, analyze and report on log data for incident response, forensics and regulatory compliance. Vendors in this space are continually improving threat intelligence and security analytics. Companies are looking to adopt this technology in order to detect threats and breaches, and by compliance needs. Early breach discovery requires effective user activity, data access and application activity monitoring.

Companies placed by Gartner in the Leaders Quadrant have been the most successful in building an installed base and establishing a revenue stream from the SIEM market. Leaders also typically have a high share of the market and high revenue growth. They've also demonstrated superior vision and execution for emerging and anticipated requirements of the market. What's more, they've garnered positive customer feedback for their SIEM products, as well as service and support of those products. 

2018

There are three new vendors (Dell, Exabearn and Securonix) and four existing vendors (IBM, Splunk, LogRhythm, McAfee) in Leaders quadrant. 

ArcSight SIEM Logger Web, Search Examples, Use Case Reports

ArcSight Logger is one of products from Micro Focus SIEM platform. It  streams real-time data and categorizes them into specific logs and easily integrates with Security Operations. As a result, organizations of any size can use this high performance log data repository to aid in faster forensic analysis of IT operations, application development, and cyber security issues, and to simultaneously address multiple regulations.

Summary

Install Latest Splunk 7.2.0 into Ubuntu and CentOS

Installing Splunk 7.2.0 into Ubuntu is super easy. I had a video to introduce how to install Splunk in a windows server in my previous post. This time, I am going to present an installation process for Splunk 7.2.0 installed into Ubuntu 18.04.  Ubuntu is running on Google Cloud Platform.

Notes: For CentOS steps and video, you can find it at the end of this post.

1. Create a new Ubuntu 18.04 LTS VM from GCP ->Computer Engine -> VM Instance

Splunk Tips and Tricks

Splunk Installation:
On Google Cloud Windows 2016 VM

On Prem Linux CentOS 7:

Configure Netflow on network devices for PRTG Netflow Monitoring

Netflow is a feature first introduced into Cisco routers and switches and then flow concept has been widely accepted by other network product vendors. Basically the network devices which support xflow feature can collect IP traffic statistics on the interfaces where xFlow is enabled, and export those statistics as xFlow records to remote defined xFlow collector.

PRTG can use this NetFlow feature for detailed bandwidth usage monitoring and it also shows you:

• where your bandwidth is used
• who is using it
• how it is being used
• why it is being used

It lets you see which specific applications are being used and how the usage might affect your network. NetFlow monitoring is included in all PRTG Network Monitor licenses, which means no special license to enable this feature. It will be counted into your sensors license.

Using PRTG SNMPv3 Monitoring Juniper SRX 240H Alarm andTemperature

One of our SRX240H is having temperature problem. Whenever the temperature reached 50 Celsius degree, system alarm will be on. Alarm email should be sent out when temperature reached threshold 50. SRX itself seems not able to send alarm email out based on this discussion. NSM or other SNMP tools may help in this situation.

PRTG is using to monitor our network devices and it works great with SNMPv3. My previous post has described how to monitor SRX's CPU, Memory, Flow Sessions etc. Alarm status and Temperature is another sensor I am looking for to monitor. There are couple of ways to do it. You can use NSM to send alarm email, firewall itself to send snmp traps to your SNMP server, or Network Monitoring Tools to pull SNMP OID values then send email. In my case, PRTG is preferred way to monitor system status and send alarming email based on the requirement.

Step 1: SNMPv3 on SRX

set snmp v3 usm local-engine user SRXAES authentication-md5 authentication-password Test1234
set snmp v3 usm local-engine user SRXAES privacy-aes128 privacy-password Test12345
set snmp engine-id local 4716
set snmp view view_all oid 1 include
set snmp filter-duplicates
set snmp health-monitor

set snmp location "<location>"
set snmp contact "<contact name>"
set snmp community <community-name> authorization read-only
set snmp community <community-name> clients <snmp-host>
set snmp community <community-name> clients 0.0.0.0/0 restrict

Archive Juniper STRM (IBM Qradar) Logs to remote server

Our Juniper STRM is running out of space after receiving more and more logs from Check Point management server and Juniper NSM. Since my STRM 500 only has about 400G storage capability and there is no other way to get budget to upgrade it to other expensive model, I decided to manually archive some older data out of this box. The steps are quite straightforward, just need to find out log folders and tar them , move them to remote ssh server.

1. Current Situation:

88% disk has been used and it is going to stop receiving the logs and flows from sources. Also reports will not be able to generated once the disk is reached certain level, about 92%.

[[email protected] ~]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2             20323232  14568560   4705628  76% /
/dev/sda3             10169544   1970636   7673988  21% /var/log
/dev/sda1               108865     30790     72454  30% /boot
tmpfs                  4091188         0   4091188   0% /dev/shm
/dev/sda8            424837936 362963724  31193700  92% /store
/dev/sda5             17263128    177164  16194896   2% /store/tmp

2. All log data are stored at /store/ariel/events folder:

Under /store/ariel/events folder, there are two sub directories payloads and records which stores all log data. The payloads is the raw data which is being sent to the STRM and the records are the normalized data which are stored onto the STRM after the parsing of the raw data.

[[email protected] events]# ls -l
total 16
drwxr-xr-x 4 root   root   4096 Jan  1 00:01 md
drwxr-xr-x 4 root   root   4096 Jan  1 00:00 payloads
drwxr-xr-x 4 root   root   4096 Jan  1 00:00 records
drwxr-xr-x 4 nobody nobody 4096 Jan  2 01:04 uncompressedCache

In both of payloads and records folders, the logs are stored by years and months.
This is the folder size for each month of 2014 under payloads folder:

This is the folder size for each month of 2014 under records folder:

3.  Tar the folder which you want to archive and move to remote ssh storage server:

cd /store/ariel/events/payload/2014 
tar -zcvf /store/tmp/2014.1.payload.tar.gz 1 
cd /store/ariel/events/records/2014 
tar -zcvf /store/tmp/2014.1.records.tar.gz 1
du -hs 2014.1.*.tar.gz
watch du -hs 2014.1.*.tar.gz

4. SCP tar files to remote site

scp /store/tmp/2014.1.*.tar.gz [email protected]:/data/backup/juniper

5. Delete files and folders

rm -rf /store/tmp/2014.1.*.tar.gz
rm -rf /store/ariel/events/payloads/2014/1
rm -rf /store/ariel/events/records/2014/1

6.  Verify

[[email protected] tmp]# df -k 
Filesystem           1K-blocks      Used Available Use% Mounted on 
/dev/sda2             20323232  14562300   4711888  76% / 
/dev/sda3             10169544   1999120   7645504  21% /var/log 
/dev/sda1               108865     30790     72454  30% /boot 
tmpfs                  4091188         0   4091188   0% /dev/shm 
/dev/sda8            424837936 329588056  73669368  82% /store 
/dev/sda5             17263128    658040  15714020   5% /store/tmp

Installation Steps of LOG Storm Free Virtual SIEM Appliance

I was reading the Top 47 Log Management Tools from ProfitBricks' blog. During quick scanning the key features and cost, I decided to give LOG Storm a try. This post is the recording steps for installation and basic configuration of this product.

Key Features: 

• In-depth threat analysis
• Flexible deployment options
• Intuitive graphical user interface
• Incident response, forensics, and discovery
• Built-in support for 1,000+ devices
• Simple device integration tool
• Reporting packs for major regulatory compliance standards
• Master console for centralized log management
• MetaRules Correlation

Cost: 

• LOG Storm Virtual SIEM Appliance: FREE
• Other deployment options and advanced solutions: Contact for a quote

Note: Free license is only for up to 5 devices and 5G storage.

1. Download

From the green "Free LOG STROM DOWNLOAD" link, you will be guided to a page with following links:

Download LOG Storm image file here.

Download LOG Storm torrent file here.

If you need to request a license key for LOG Storm, please click here.

Click the image file the downloading will automatically started. You will get a 1.39G LOG_Storm_4.5.0.20_Eval_VA.ova file.

2. Import OVA into VM lab environment

Double click the downloaded ova file, VM Workstation will import it into your default Virtual Machine folder.

Default vm setting for LOG Storm is using 6GB memory. I changed it to 4GB and it is still working fine in my lab environment.

3. Start your VM 

Default user name/password is htadmin/htadmin
You will have to accept the agreement, change the htadmin password, do basic network and information configuration. Then wait at most 5 minutes to let virtual appliance to configure itself based on your input.

4. First SSH Log in

After virtual appliance rebooted, use SSH log into system with htadmin username.

After logged into system, it will ask you to enter valid license you got from the email.


Linux logstorm 2.6.32-5-amd64 #1 SMP Tue May 13 16:34:35 UTC 2014 x86_64
Last login: Tue Jan  6 11:02:13 2015
Do you need to change your configuration before entering your license? ([Y]es, [N]o, Enter = , '?' for help) : N
Please enter your LOG Storm appliance license (what you enter will NOT be echoed back to you): ('help' for help) : 
License is valid

Activating LOG Storm services

 From the main menu, you will need to select 2. Password Management to set Admin Account Password which will be used to log into WebUI

5. WebUI Log in

Using your browser to open https://<Virtual Appliance IP address>, you will get following screenshot.

 Click 'Launch Client'

 Enter Admin username and password

 Now it is the dashboard for your SIEM Virtual Appliance.

6. Reference

License Request Link Here

• Step 1: Download the FREE LOG Storm Virtual SIEM Appliance
• Step 2: View the LOG Storm Virtual SIEM Appliance Data Sheet
• Step 3: View the LOG Storm Installation and Activation Video
• Step 4: View the Getting Started Demonstrations

Additional Demonstrations:

• View the LOG Storm Reporting Features
• View the LOG Storm Raw Message Log Functionality
• View the LOG Storm Device Builder Tutorial
• View the LOG Storm Rules Creation Tutorial

Forwarding Checkpoint Management Server Firewall logs to an external syslog server STRM/Qradar SIEM

There are two ways to integrate STRM with Check Point Firewalls devices.

1. Using Syslog

On Check Point management station, you can follow these steps to redirect firewall logs and audit logs to the external syslog server:

a) Vi /etc/syslog.conf, on the management station, and add the following line at the end of the file:

local5.info @hostname

such as :

          local5.info @10.94.20.23

where ‘10.94.20.23’ is the IP of the syslog server (Juniper STRM).

b) if your management server is SecurePlatform - Execute ‘service syslog restart’.

c) Add this command to /etc/rc.d/init.d/cpboot:

fw log -ftnl | logger -p local5.info -t Firewall &

or 

forward audit log to external syslog server by add following command:

fw log -ftnl $FWDIR/fw.adtlog | awk 'NF' | logger -p local5.info -t Firewall_Audit &

d) reboot Checkpoint management server and configure a new log source in STRM. Deploy Changes to STRM as well. 

e) Verify:

tcpdump host 10.94.20.23

[[email protected]]# tcpdump host 10.94.20.23
tcpdump: listening on Mgmt
12:54:18.534293 CP-Management.syslog > 10.94.20.23.syslog: udp 253 (DF)
12:54:18.538859 CP-Management.syslog > 10.94.20.23.syslog: udp 16 (DF)
12:54:18.539622 CP-Management.syslog > 10.94.20.23.syslog: udp 225 (DF)
12:54:18.540382 CP-Management.syslog > 10.94.20.23.syslog: udp 16 (DF)
12:54:18.541115 CP-Management.syslog > 10.94.20.23.syslog: udp 252 (DF)
12:54:18.541904 CP-Management.syslog > 10.94.20.23.syslog: udp 16 (DF)
12:54:20.536629 CP-Management.syslog > 10.94.20.23.syslog: udp 280 (DF)
12:54:20.538424 CP-Management.syslog > 10.94.20.23.syslog: udp 16 (DF)
12:54:20.539194 CP-Management.syslog > 10.94.20.23.syslog: udp 228 (DF)
12:54:20.540009 CP-Management.syslog > 10.94.20.23.syslog: udp 16 (DF)
12:54:22.539075 CP-Management.syslog > 10.94.20.23.syslog: udp 225 (DF)
12:54:22.543184 CP-Management.syslog > 10.94.20.23.syslog: udp 16 (DF)
12:54:28.540703 CP-Management.syslog > 10.94.20.23.syslog: udp 249 (DF)
12:54:28.543712 CP-Management.syslog > 10.94.20.23.syslog: udp 16 (DF)
12:54:28.544410 CP-Management.syslog > 10.94.20.23.syslog: udp 225 (DF)
12:54:28.545036 CP-Management.syslog > 10.94.20.23.syslog: udp 16 (DF)

On STRM server 10.94.20.23, you should be able to see following logs activities:

2. Using OPSEC / LEA

a. Creating an OPSEC Application Object from Servers and OPSEC tab:

In my lab, STRM_10.94.200.23 created.

Note: Communication Initialized but trust not established, it is still fine to do firewall policy push. The communication will be established by itself after STRM configuration part done.

b. Write Down and Copy Two SIC DN info for STRM configuration

One is from new created OPSEC application: STRM_10.94.200.23
CN=STRM_10.94.200.23,O=CP-Management..wtx8w4

Another is from Mgmt Server CP_Management as show below:

cn=cp_mgmt,o=CP-Management..wtx8w4

c. STRM Log Source

 d. add a new Checkpoint Firewall-1 OPSEC / LEA log Source

e. Verify SIC Connection from Checkpoint Mgmt Server OPSEC Application STRM_10.94.200.23

Reference:

1.  Juniper STRM Configuring DSMs
2.  How to send FireWall logs from Gaia-based Security Management Server to an external Syslog server
3.  Forward Logs from Checkpoint SmartCenter Management Server and Juniper NSM / IDP to Syslog Server

Forward Logs from Checkpoint SmartCenter Management Server and Juniper NSM / IDP to Syslog Server

Two KBs regarding how to collect log from Checkpoint and Juniper:

1. Configuring SmartCenter to send logs to syslog server

Solution ID: sk33423

Proceed as follows:

a. On the SmartCenter server edit the /etc/syslog.conf file and add the following line:

local4.info <TAB> @IP_OF_REMOTE_BOX

b. Add the following line to the end bottom of /etc/rc.d/init.d/cpboot file, to be executed on boot up:

fw log -ftnl 2> /dev/null | awk 'NF' | logger -p local4.info -t Firewall &

Notes:

The '&' in the command syntax ensures that this command runs in the background. If the '&' is not included in the command, the OS stops at loading the syslogd service and you never get a login prompt at the console.
For more information about the fw log command, refer to the R75 Command Line Interface (CLI) Reference Guide.

c. Reboot.
Note: cpstop/cpstart is insufficient to make this work.

2.NSM can forward NSM logs as well as device traffic logs via syslog, SNMP, e-mail or even a custom script.

You need to define this in "Action Manager" from the NSM GUI client.

Check this KB article:
http://kb.juniper.net/KB11810

NSM Administration Guide the chapter "Forwarding Logs":
http://www.juniper.net/techpubs/software/management/security-manager/

Forward logs from NSM to a Syslog server

[KB11810] Show KB Properties

---

1. Login to NSM GUI


2. Go to "Action Manager" and click  "Action Parameters"


3. Fill in the Syslog server IP address and the Syslog facility that NSM will categorize the logs as.


4. Click "OK"

This informs NSM that an external Syslog server is available for use.  Two mode are available to forward logs to Syslog.

Device Log Action Criteria Mode:   Located under the "action manager", this mode allows defining a global logging criteria for all devices in a domain.
The criteria can be based on category, sub-category and severity and will apply to all logs received.

Policy Manager Mode:  Allows finer control on which traffic log will be forwarded to Syslog by adding the "Log action" to the desired rule options.   This allows forwarding of traffic logs to Syslog only for the desired rules.    Enable "Syslog" under "Log/Count" rule options for each rule.

3. IDP Appliance 

To configure Juniper IDP Appliance to send syslogs to STRM (IP Adress of STRM is assumed to be 172.19.47.201)

Login to the NSM system that is managing the IDP

1. Edit the IDP device and go to Report Settings
2. Configure the Syslog server as shown below and update the device