Understanding Splunk Monitoring : BLOG 1

This blog is nothing but my notes from Splunk admin training available on Linkedin Learning by Josh Samuelson , System Admin and DevOps Engineer.

This will be a series of 5 blogs :-

1. Understanding Splunk 

2. Search 

3. Reporting and Dashboard

4. Alerting

5. Practical Splunk 

So we use Splunk to monitor machine data and system logs.

What we can do with System Logs of Splunk : 

Collect , Search , Reporting , Visualizations and Alerting.

Collect : One of the major benefit of splunk is that , you can collect all of your data . No need to hunt for logs on multiple server and also gives you benefit of cross reference. Tracing issue from App server to Database server.

Search : A powerful way of searching which comes up with simple filtering and basic constraints along with advance Search context

Reporting Visualization and Alerting : Provides you with the functionality of generating simple reports and dashboards . Dashboards are nothing but data abstracted into simple graphs which helps you to understand pattern and reach to the root cause of any issue quickly and easily. Alerting is just setting the thresholds for which you need your team to be alerted which reduces the monitoring efforts.

Few Splunk Commands that is good to have :-

At the location : /opt/splunk/bin 

1. ./splunk start :- This command is used to start the splunk tool , at first go you need to read the terms and conditions and provide admin user and password when prompted. ]

2. ./splunk enable boot-start : Configure Splunk Enterprise to start at boot time

3. systemctl enable splunk : The enable command serves for executing the service since the initialization if consists of one or more units or unit instances. This will create a set of symlinks, as encoded in the [Install] sections of the indicated unit files. the system manager configuration is reloaded (in a way equivalent to daemon-reload), in order to ensure the changes are taken into account immediately. ) [SRC : GFG]  

Example : sudo systemctl enable name_service.service

4.systemctl enable splunk : The command start serves for starting (activate) one or more units specified on the command line.

Check Ip on which splunk is hosted ?

   You can get the IP on which splunk is hosted using ifconfig , and after that open any browser htttp://<IP on which splunk is being hosted>:8000.

    You can login in splunk using admin name and password that you must have filled during installation , In case of installation was done by someone else . You can always ask the admin to create an user for you.

Adding data to Splunk 

Before monitoring or creating alert for anything you need to add details of host for that you will see options here 

We have different options that we have to add in splunk 

 

Setting up splunk user ?

Once you are able to login , you need to go to see the setting and explore the options.

1. Users : will show all the different users that are present in the splunk account at that point of time.

You can see the green button "NEW USER"  in which you can add new user :-

Apart from role others are just basic or personal information that are required .

Let's discuss about the role.

Role : Role is just composite name assigned to a set of capabilities , which is nothing but action that can be performed by an user.

To create role we have two options 

1. Create a role based on the user being created :-

In the above image you can see that , you have an checkbox which say Create a role for this user. 

When you select multiple role for an user and you understand that the same set of role can be used in future for other user . you can make a role which has other roles in it.

E.g: You can assign admin, power, splunk-system-role , and other to one user and create a role which will have all these 3 already in one when creating new user.

2. Create a new role :- 

Go to Settings >  User and authentication > Roles

a. Inherit : You can inherit capabilities from an existing role 

b. Capabilities : You can add single single capabilities.

c. Indexes : Enable both the "Included" and "Default" checkboxes for an index to make that index searchable by default for this role. You must save this role before you can see its inherited wildcards.

d. Restrictions 

e. Resources

Apart from this you have option of 

Token : To set token based authentication .

Authentication Methods: You can add these different authentication method to make sure that you are using a protected and easy to use method , with the option of adding multifactor authenetication.

 

Password Policy Management : In this option you will have options to create a password policy which majorly depends on your organization and needs to be set as per the agreement they want to have.

Understanding the Search feature

You can directly search  in this tab and the result shows is split in time and event , which helps you have a better look.

with added feature like host from which that particular result is coming up , Source is the log file  and Source type is the log from where it is coming.

Diving in the Report Tab :-

Report tab already has a handful of reports that are used generally , reports is the filtered search data that make some sense and fulfill some agenda.

to understand it more clearly use open in search  option.

you are just filtering a search making and adding up time constraint to make some useful monitoring insights.

Connection between Search and Report 

What this particular report is doing is just searching Error failed or severe.

Manipulating the report feature :-

It is recommended that you never use the actual report that is initially present you have the option of clone in which you can perform the changes as per the customizations required.

You can clone change the description and title and work on the cloned report.

Edit Options 

Edit Schedule 

Add-Ons 

How to add addons in splunk can be understood in below slides , from adding these you can just utilize features that are their performing a small installation.

 

You can search and explore your options here 

You need to provide ID and Password for more understanding and then it will be in your system.

We will be giving you more blogs for splunk in coming future , Stay Tuned and let us know which topics related to splunk you need more information about.