Why do we need separate to our logging mechanism in microservice architecture?

You have to make sure if you need it. Yes, you can use it if you want to manage your logs at one point and you don’t want to do HTTP requests for each log line.

Architechtural Diagram

What are the benefits of this architecture?

Filebeat can listen to all microservices that in same network.
Filebeat ships the collected logs as a stream. So it increases the productivity of data transfer.
You don’t need do new implementations. When you add new component into the docker-compose.yml, it will work for the component too.
Elasticsearch is one of the most usable database for huge data storing.
You can monitor metrics and graphics on Kibana.

Filebeat is listening to all containers that are implemented and send to ELK stack. You can configure it via filebeat.yml file.

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

output.logstash:
  hosts: ["logstash:5000"]

logging.level: error

According to the above filebeat.yml file, the consumed logs are sending to logstash. If you don’t want to use Logstash for filtering, you can write “elasticsearch” as output.

output.elasticseach:
  hosts: ["elasticsearch:5000"]

So, What kind of docker-compose.yml file you need ?

version: '3.1'

services:
  X-service:
    build: ./X-service
    ports:
      - "8080:8080"

logstash:
  container_name: logstash
  image: logstash:7.5.0
  volumes:
    - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    - ./logstash.template.json:/usr/share/logstash/templates/logstash.template.json
  ports:
    - 5044:5044
  depends_on:
    - elasticsearch
  stdin_open: true
  logging:
    driver: "json-file"
    options:
      max-size: "10m"
      max-file: "50"

elasticsearch:
  image: elasticsearch:8.0.1
  volumes:
    - elasticsearch:/usr/share/elasticsearch/data:z
  ports:
    - "9200:9200"
  environment:
    ES_JAVA_OPTS: -Xmx256m -Xms256m
    discovery.type: single-node
    xpack.security.enabled: false
    logger.level: ERROR

kibana:
  image: kibana:8.0.1
  ports:
    - "5601:5601"

filebeat:
    image: elastic/filebeat:8.0.1
    user: root
    volumes:
      - ./filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - type: bind
        source: /var/lib/docker/containers
        target: /var/lib/docker/containers
        read_only: true
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
        read_only: true    depends_on:
      - logstash

Volume paths and configs can be symbolic in the above code blocks. You can customize them according to your design. Also, you need to add volumes for components that are required.

Poorly designed code usually takes more code to do the same things, often because the code quite literally does the same thing in several places.

Martin Fowler

Thanks for your time!

Upcoming events

The Test Automation Meetup
PLEASE RSVP SO THAT WE KNOW HOW MUCH FOOD WE WILL NEED Test automation is a cornerstone of effective software development. It's about creating robust, predictable test suites that enhance quality and reliability. By diving into automation, you're architecting systems that ensure consistency and catch issues early. This expertise not only improves the development process but also broadens your skillset, making you a more versatile team member. Whether you're a developer looking to enhance your testing skills or a QA professional aiming to dive deeper into automation, RSVP for an evening of learning, delicious food, and the fusion of coding and quality assurance! 🚀🚀 18:00 – 🚪 Doors open to the public 18:15 – 🍕 Let’s eat 19:00 – 📢 First round of Talks 19:45 – 🍹 Small break 20:00 – 📢 Second round of Talks 20:45 – 🍻 Drinks 21:00 – 🙋‍♀️ See you next time? First Round of Talks: The Power of Cross-browser Component Testing - Clarke Verdel, SR. Front-end Developer at iO How can you use Component Testing to ensure consistency cross-browser? Second Round of Talks: Omg who wrote this **** code!? - Erwin Heitzman, SR. Test Automation Engineer at Rabobank How can tests help you and your team? Beyond the Unit Test - Christian Würthner, SR. Android Developer at iO How can you do advanced automated testing for, for instance, biometrics? RSVP now to secure your spot, and let's explore the fascinating world of test automation together!
December 11 | Coven of Wisdom - Amsterdam
Go to page for The Test Automation Meetup
Coven of Wisdom - Herentals - Winter `24 edition
Worstelen jij en je team met automated testing en performance? Kom naar onze meetup waar ervaren sprekers hun inzichten en ervaringen delen over het bouwen van robuuste en efficiënte applicaties. Schrijf je in voor een avond vol kennis, heerlijk eten en een mix van creativiteit en technologie! 🚀 18:00 – 🚪 Deuren open 18:15 – 🍕 Food & drinks 19:00 – 📢 Talk 1 20:00 – 🍹 Kleine pauze 20:15 – 📢 Talk 2 21:00 – 🙋‍♀️ Drinks 22:00 – 🍻 Tot de volgende keer? Tijdens deze meetup gaan we dieper in op automated testing en performance. Onze sprekers delen heel wat praktische inzichten en ervaringen. Ze vertellen je hoe je effectieve geautomatiseerde tests kunt schrijven en onderhouden, en hoe je de prestaties van je applicatie kunt optimaliseren. Houd onze updates in de gaten voor meer informatie over de sprekers en hun specifieke onderwerpen. Over iO Wij zijn iO: een groeiend team van experts die end-to-end-diensten aanbieden voor communicatie en digitale transformatie. We denken groot en werken lokaal. Aan strategie, creatie, content, marketing en technologie. In nauwe samenwerking met onze klanten om hun merken te versterken, hun digitale systemen te verbeteren en hun toekomstbestendige groei veilig te stellen. We helpen klanten niet alleen hun zakelijke doelen te bereiken. Samen verkennen en benutten we de eindeloze mogelijkheden die markten in constante verandering bieden. De springplank voor die visie is talent. Onze campus is onze broedplaats voor innovatie, die een omgeving creëert die talent de ruimte en stimulans geeft die het nodig heeft om te ontkiemen, te ontwikkelen en te floreren. Want werken aan de infinite opportunities van morgen, dat doen we vandaag.
December 12 | Coven of Wisdom Herentals
Go to page for Coven of Wisdom - Herentals - Winter `24 edition
Mastering Event-Driven Design
PLEASE RSVP SO THAT WE KNOW HOW MUCH FOOD WE WILL NEED Are you and your team struggling with event-driven microservices? Join us for a meetup with Mehmet Akif Tütüncü, a senior software engineer, who has given multiple great talks so far and Allard Buijze founder of CTO and founder of AxonIQ, who built the fundaments of the Axon Framework. RSVP for an evening of learning, delicious food, and the fusion of creativity and tech! 🚀 18:00 – 🚪 Doors open to the public 18:15 – 🍕 Let’s eat 19:00 – 📢 Getting Your Axe On Event Sourcing with Axon Framework 20:00 – 🍹 Small break 20:15 – 📢 Event-Driven Microservices - Beyond the Fairy Tale 21:00 – 🙋‍♀️ drinks 22:00 – 🍻 See you next time? Details: Getting Your Axe On - Event Sourcing with Axon Framework In this presentation, we will explore the basics of event-driven architecture using Axon Framework. We'll start by explaining key concepts such as Event Sourcing and Command Query Responsibility Segregation (CQRS), and how they can improve the scalability and maintainability of modern applications. You will learn what Axon Framework is, how it simplifies implementing these patterns, and see hands-on examples of setting up a project with Axon Framework and Spring Boot. Whether you are new to these concepts or looking to understand them more, this session will provide practical insights and tools to help you build resilient and efficient applications. Event-Driven Microservices - Beyond the Fairy Tale Our applications need to be faster, better, bigger, smarter, and more enjoyable to meet our demanding end-users needs. In recent years, the way we build, run, and operate our software has changed significantly. We use scalable platforms to deploy and manage our applications. Instead of big monolithic deployment applications, we now deploy small, functionally consistent components as microservices. Problem. Solved. Right? Unfortunately, for most of us, microservices, and especially their event-driven variants, do not deliver on the beautiful, fairy-tale-like promises that surround them.In this session, Allard will share a different take on microservices. We will see that not much has changed in how we build software, which is why so many “microservices projects” fail nowadays. What lessons can we learn from concepts like DDD, CQRS, and Event Sourcing to help manage the complexity of our systems? He will also show how message-driven communication allows us to focus on finding the boundaries of functionally cohesive components, which we can evolve into microservices should the need arise.
September 19 | Coven of Wisdom - Utrecht
Go to page for Mastering Event-Driven Design

Separated Logging with Filebeat and ELK stack

What are the benefits of this architecture?

Upcoming events

The Test Automation Meetup

Coven of Wisdom - Herentals - Winter `24 edition

Mastering Event-Driven Design

Share

Jobs

Senior Web Developer

.Net Backend Developer / Software Engineer

Lead Developer

Drupal Developer

View all jobs