1. Set up
I might advocate deploying Graylog utilizing Docker containers. Graylog gives an instance docker compose file that already specifies which pictures to drag and which ports are required. This compose file will create three companies, a MongoDB database for storing Graylog configuration, a Information Node which handles looking, and the Graylog utility itself which exposes an internet consumer interface (accesible from localhost:9000). Moreover, the compose file creates the required networks and volumes for persistent knowledge storage.
You could find the instance compose file supplied by Graylog on this Github repository: https://github.com/Graylog2/docker-compose/blob/main/open-core/docker-compose.yml
2. Configuration
There are two surroundings variables that you’ll want to set within the compose file earlier than launching Graylog: GRAYLOG_PASSWORD_SECRET and GRAYLOG_ROOT_PASSWORD_SHA2. The primary worth might be any string. The second worth is specifically difficult, because it must be the SHA-256 checksum of the password you’ll sort into the Graylog internet interface to log in because the admin consumer. The code under reveals methods to get the SHA-256 checksum for a given password.
# Keep in mind the string "YourPassword"! That's the worth you'll want to
# log into the Graylog internet interface
echo -n YourPassword | shasum -a 256
I like to recommend storing these values in a separate .env file.
# .env
GRAYLOG_PASSWORD_SECRET=your_password
GRAYLOG_ROOT_PASSWORD_SHA2=your_password_hash
In the event you observe this suggestion, you’ll want to add the next traces within the compose.yaml file.
companies:
datanote:
env_file:
- "path/to/your/.env/file"
...
graylog:
env_file:
- "path/to/your/.env/file"
...
...
Lastly, in case you are working this setup in a digital machine, you’ll want to make sure that it has entry to sufficient reminiscence. For instance, in case you are utilizing Home windows Subsystem for Linux (WSL), you’ll want to run this command earlier than launching the docker containers.
sudo sysctl -w vm.max_map_count=262144
Now you’re able to provoke the Graylog utility. Run the command under from the identical listing the place the compose.yaml file is saved to launch Graylog.
docker compose up
Take note of the output it generates. The message under will present up in the midst of the output and it incorporates essential data.
Open an internet browser and go localhost:9000. There you need to carry out a one-time setup. Log into the Graylog internet utility utilizing the username and password supplied within the terminal and create a Certificates Authority (observe the directions within the internet utility). When completed, click on the “Resume startup” button.
After this one-time setup, it is best to see the default Graylog authentication display screen. This time, use the password that you just saved within the .env file. Be certain to make use of its authentic kind (not the hashed worth). For instance, if the password was “Hello” and the hash was “2359B2”, enter “Hello”.
3. Getting logs into Graylog
To ingest logs into Graylog, we first have to create an Enter (a listener with a specific configuration).
Choose the System tab within the higher navigation bar and choose the choice Inputs from the dropdown menu. Then launch a brand new enter of sort GELF TCP (GELF stands for Graylog Prolonged Log Format).
Enter any title you favor, and set the Bind deal with to 0.0.0.0. Be certain to allow TLS connection (image under). Go away the remainder of the choices with their default values.
Now Graylog is listening for TCP messages in port 12201. Nonetheless we’re not sending any messages to that port but.
Subsequently, we are going to create a python script to ahead messages from a Kafka dealer and subject to localhost:12201. There are numerous alternative routes of attaining this. We are going to make use of the logging , kafka , andgraypy python libraries.
First we have to set up a reference to the Kafka dealer. You should use the next command to take action.
ssh -L :localhost: @ -NTf
Then, run the script to learn the messages out of your desired Kafka subject. The instance supplied assumes you established the tunnel with Kafka on the native port 9092 and can learn the most recent incoming messages (it is not going to learn all messages from the beginning of the subject).
from argparse import ArgumentParser
from kafka import KafkaConsumer
from graypy import GELFTLSHandler
from logging import getLogger, DEBUG
from time import sleepdef fundamental(subject: str):
shopper = KafkaConsumer(
subject,
bootstrap_servers="localhost:9092",
auto_offset_reset="newest",
group_id=subject,
enable_auto_commit=True,
auto_commit_interval_ms=1000,
value_deserializer=lambda x: x.decode('utf-8')
)
whereas not shopper.task():
print("Ready for partition task...")
shopper.ballot(5000)
print("Partition assigned")
shopper.seek_to_end()
handler = GELFTLSHandler(host='localhost', port=12201)
logger = getLogger('kafka-test')
logger.setLevel(DEBUG)
logger.addHandler(handler)
for log in shopper:
message = log.worth
logger.debug(message)
if __name__=='__main__':
argparser = ArgumentParser()
argparser.add_argument('-t', dest='subject', sort=str)
args = argparser.parse_args()
fundamental(subject = args.subject)
You must now be receiving incoming messages. Choose the Search hyperlink within the Graylog utility navigation bar. You must see one thing much like the picture under.
4. Processing logs on Graylog
Lastly, we are going to separate the score logs from the remainder of the incoming messages. To realize this, we are going to use Graylog’s routing capabilities. Primarily, we are going to use “streams” and “strem guidelines” to isolate any such logs.
Lets begin by creating a brand new index the place the rankings logs will likely be saved after being separated from the remaining. Use the navigation bar to go System -> Indices and click on on the button “Create index set”. You may choose any of the Constructed-in Templates. Then present a title, description and prefix of your selection in your new index, and choose “Create index set” on the backside of the web page.
Then, use the navigation bar to go the Streams web page throughout the Graylog utility. Right here, be sure to pick the brand new index you created within the earlier step and (optionally) take away the matching logs from the “Default Stream” to keep away from storing duplicates.
Lastly, we’re going to create a “stream rule” by clicking the “Information Rounting” button subsequent to your new “stream” after which deciding on “Create Rule”
We are going to use this rule to filter logs primarily based on the content material of their messages utilizing common expressions.
After creating the “stream rule”, click on the play button subsequent to our new “Rankings” stream within the “Streams” web page.
Again within the “Search” web page, it is best to now have the ability to visualize solely the logs that fulfilled our “stream rule” and are being dealt with by the “Rankings” stream (be sure to hit enter after deciding on the stream to run the question).
On the decrease left panel (“All Messages”), all of the messages proven ought to match the common expression supplied.
