The Rocket-fast Syslog Server - Rsyslog Client and Server Configuration - NETSEC

Latest

Learning, Sharing, Creating

Cybersecurity Memo

Sunday, November 17, 2019

The Rocket-fast Syslog Server - Rsyslog Client and Server Configuration

Rsyslog is an Open Source logging program, which is the most popular logging mechanism in a huge number of Linux distributions. It's also the default logging service in CentOS 7 or RHEL 7. Rsyslog daemon in CentOS can be configured to run as a server in order collect log messages from multiple network devices. In this post, I am using two CentOS7 linux machines to test Rsyslog as server and client.

Topology

Client machine 34.67.242.159 will send out local logs to remote central syslog server 35.224.49.121.
Both machines are running on CentOS7.




Related YouTube Video:

Rsyslog Server Installation and Configuration

1. Install Rsyslog 


[jon_netsec@rsyslog-server1 ~]$ 
[jon_netsec@rsyslog-server1 ~]$ sudo -i
[root@rsyslog-server1 ~]# sudo yum update && yum install rsyslog 
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos4.zswap.net
 * epel: mirror.colorado.edu
 * extras: centos4.zswap.net
 * updates: centos4.zswap.net
No packages marked for update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos4.zswap.net
 * epel: mirror.uic.edu
 * extras: centos4.zswap.net
 * updates: centos4.zswap.net
Package rsyslog-8.24.0-41.el7_7.2.x86_64 already installed and latest version
Nothing to do
[root@rsyslog-server1 ~]# systemctl start rsyslog
[root@rsyslog-server1 ~]# systemctl enable rsyslog
[root@rsyslog-server1 ~]# systemctl status rsyslog
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-11-15 02:32:14 UTC; 11h ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 17303 (rsyslogd)
   CGroup: /system.slice/rsyslog.service
           └─17303 /usr/sbin/rsyslogd -n

Nov 15 02:32:14 rsyslog-server1 systemd[1]: Starting System Logging Service...
Nov 15 02:32:14 rsyslog-server1 rsyslogd[17303]:  [origin software="rsyslogd" swVersion="8.24....rt
Nov 15 02:32:14 rsyslog-server1 systemd[1]: Started System Logging Service.
Hint: Some lines were ellipsized, use -l to show in full.
[root@rsyslog-server1 ~]# vim /etc/rsyslog.conf
[root@rsyslog-server1 ~]# vim /etc/rsyslog.conf
[root@rsyslog-server1 ~]# sysmtemctl restart rsyslog
-bash: sysmtemctl: command not found
[root@rsyslog-server1 ~]# systemctl restart rsyslog
[root@rsyslog-server1 ~]# ss -tulnp | grep "rsyslog"
udp    UNCONN     0      0         *:514                   *:*                   users:(("rsyslogd",pid=2507,fd=3))
udp    UNCONN     0      0      [::]:514                [::]:*                   users:(("rsyslogd",pid=2507,fd=4))
tcp    LISTEN     0      25        *:514                   *:*                   users:(("rsyslogd",pid=2507,fd=5))
tcp    LISTEN     0      25     [::]:514                [::]:*                   users:(("rsyslogd",pid=2507,fd=6))


[root@rsyslog-server1 ~]# setenforce Permissive
[root@rsyslog-server1 ~]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31
[root@rsyslog-server1 ~]# getenforce
Permissive
[root@rsyslog-server1 ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

[root@rsyslog-server1 ~]# systemctl stop firewalld


Note: You can disable firewall and selinux feature as I did. The better way is to configure selinux and firewall to allow udp /tcp 514 traffic. In my lab, I just quickly disabled them to show you how Rsyslog is working.




2. Configure Rsyslog as server to collect all log/remote logs

By default, rsyslog uses the imjournal and imusock modules for importing structured log messages from systemd journal and for accepting syslog messages from applications running on the local system via Unix sockets, respectively.

To configure rsyslog as a network/central logging server, you need to set the protocol (either UDP or TCP or both) it will use for remote syslog reception as well as the port it listens on.

If you want to use a UDP connection, which is faster but unreliable, search and uncomment the lines below for udp.

To use TCP connection (which is slower but more reliable), search and uncomment the lines below for tcp.

[root@rsyslog-server1 ~]# vim /etc/rsyslog.conf

...

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514


$template RemoteLogs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
& ~

#### GLOBAL DIRECTIVES ####
...

[root@rsyslog-server1 ~]# systemctl restart rsyslog

Next, you need to define the ruleset for processing remote logs in the following format. Looking at the above ruleset template, the first rule is “$template RemoteLogs,”/var/log/%HOSTNAME%/%PROGRAMNAME%.log””.

The directive $template tells rsyslog daemon to gather and write all of the received remote messages to distinct logs under /var/log, based on the hostname (client machine name) and remote client facility (program/application) that generated the messages as defined by the settings present in the template RemoteLogs.

The second line “*.* ?RemoteLogs” means record messages from all facilities at all severity levels using the RemoteLogs template configuration.

The final line “& ~” instructs rsyslog to stop processing the messages once it is written to a file. If you don’t include “& ~”, messages will instead be be written to the local files.

Another template example:

 $template DailyPerHost,"/logs/syslog_devices/%FROMHOST-IP%/%FROMHOST-IP%-%$YEAR%-%$MONTH%-%$DAY%.log"
 *.* -?DailyPerHost

RsyslogClient Installation and Configuration

1. Install Rsyslog 


[jon_netsec@rsyslog-client1 ~]$ curl ifconfig.me
34.67.242.159[jon_netsec@rsyslog-client1 ~]$ 
[jon_netsec@rsyslog-client1 ~]$ 
[jon_netsec@rsyslog-client1 ~]$ sudo yum update && yum install rsyslog 
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos4.zswap.net
 * epel: mirror.grid.uchicago.edu
 * extras: centos4.zswap.net
 * updates: centos4.zswap.net
No packages marked for update
Loaded plugins: fastestmirror
You need to be root to perform this command.
[jon_netsec@rsyslog-client1 ~]$ sudo -i
[root@rsyslog-client1 ~]# sudo yum update && yum install rsyslog 
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos4.zswap.net
 * epel: mirror.grid.uchicago.edu
 * extras: centos4.zswap.net
 * updates: centos4.zswap.net
No packages marked for update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos4.zswap.net
 * epel: mirror.grid.uchicago.edu
 * extras: centos4.zswap.net
 * updates: centos4.zswap.net
Resolving Dependencies
There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help).
The program yum-complete-transaction is found in the yum-utils package.
--> Running transaction check
---> Package rsyslog.x86_64 0:8.24.0-41.el7_7 will be updated
---> Package rsyslog.x86_64 0:8.24.0-41.el7_7.2 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================
 Package             Arch               Version                       Repository           Size
================================================================================================
Updating:
 rsyslog             x86_64             8.24.0-41.el7_7.2             updates             616 k

Transaction Summary
================================================================================================
Upgrade  1 Package

Total size: 616 k
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test


Transaction check error:
  package rsyslog-8.24.0-41.el7_7.2.x86_64 is already installed

Error Summary
-------------

[root@rsyslog-client1 ~]# systemctl start rsyslog
[root@rsyslog-client1 ~]# systemctl enable rsyslog
[root@rsyslog-client1 ~]# systemctl status rsyslog
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-11-15 02:24:26 UTC; 12h ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/

 Main PID: 905 (rsyslogd)
   CGroup: /system.slice/rsyslog.service
           └─905 /usr/sbin/rsyslogd -n

Nov 15 02:24:26 rsyslog-client1 systemd[1]: Starting System Logging Service...
Nov 15 02:24:26 rsyslog-client1 rsyslogd[905]:  [origin software="rsyslogd" swVersion="8.24...rt
Nov 15 02:24:26 rsyslog-client1 systemd[1]: Started System Logging Service.
Hint: Some lines were ellipsized, use -l to show in full.



2. Configure Rsyslog as client to collect local log/remote logs to send to remote Rsyslog server

To force the rsyslog daemon to act as a log client and forward all locally generated log messages to the remote rsyslog server, add this forwarding rule, at the end of the file as shown in the following screenshot.
*. *  @@35.224.49.121:514

[root@rsyslog-client1 ~]# vim /etc/rsyslog.conf 

...
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
*.* @@35.224.49.121:514



[root@rsyslog-client1 ~]# systemctl restart rsyslog
[root@rsyslog-client1 ~]# logger -s -p user.info Testing Rsyslog Client log
jon_netsec: Testing Rsyslog Client log
[root@rsyslog-client1 ~]# 



Testing and Verifying



[root@rsyslog-server1 log]# ls
audit     cron       grubby_prune_debug  ntpstats         secure    wtmp
boot.log  dmesg      lastlog             qemu-ga          spooler   yum.log
btmp      firewalld  maillog             rsyslog-client1  tallylog
chrony    grubby     messages            rsyslog-server1  tuned
[root@rsyslog-server1 log]# cd rsyslog-server1/
[root@rsyslog-server1 rsyslog-server1]# ls
jon_netsec.log  kernel.log  polkitd.log  rsyslogd.log  sshd.log  systemd.log
[root@rsyslog-server1 rsyslog-server1]# cat jon_netsec.log 
2019-11-15T14:50:52.794133+00:00 rsyslog-server1 jon_netsec: Testing Server Rsyslog
[root@rsyslog-server1 rsyslog-server1]# cd ..
[root@rsyslog-server1 log]# cd rsyslog-client1/
[root@rsyslog-server1 rsyslog-client1]# ls
dbus.log      jon_netsec.log      nm-dispatcher.log  rsyslogd.log
dhclient.log  NetworkManager.log  polkitd.log        systemd.log
[root@rsyslog-server1 rsyslog-client1]# cat jon_netsec.log 
2019-11-15T14:51:40+00:00 rsyslog-client1 jon_netsec: Testing Rsyslog Client log
[root@rsyslog-server1 rsyslog-client1]# vim /etc/rsyslog.conf
[root@rsyslog-server1 rsyslog-client1]# 







No comments:

Post a Comment