Hybrid Cloud 012 - Observability, Fluentd and Fluent Bit with Calyptia

In this episode, Chris chats to Anurag Gupta, co-founder at Calyptia about the requirements for system observability, specifically using Fluentd and Fluent Bit. Fluentd is a data collection tool, written in Ruby and C and used to collect data from multiple system types. Fluent Bit is a log collection tool written in C with an emphasis on efficiently collecting data from embedded Linux systems, including the IoT market. In this discussion Anurag explains the background to observability versus monitoring and how modern SaaS applications also need data to be collected and analysed. Data collection needs to be secure, filtered and obfuscated where necessary, to ensure PII or other sensitive data isn’t leaked inadvertently. Calyptia manages Fluentd and Fluent Bit deployments, providing a SaaS portal to establish rules and policies on data sources, filtering and storage.

For more information, check out https://calyptia.com/ and https://www.fluentd.org/.

Elapsed Time: 00:41:02

Timeline

00:00:00 – Intros
00:02:40 – Microsoft loves Linux! (Now)
00:04:13 – What is Fluentd and Fluent Bit?
00:06:23 – Modern app complexity requires a greater depth – observability
00:08:30 – A filter system is required to highlight important metrics
00:09:16 – What data are customers typically collecting?
00:11:30 – SaaS applications also have observability requirements
00:13:58 – Why Fluentd and Fluent Bit?
00:16:08 – How much overhead does observability add?
00:19:12 – Fluentd/Fluent Bit are light touch solutions
00:19:40 – How are policies applied to data collection?
00:22:00 – Calyptia adds on enterprise-class management functionality
00:25:15 – Log data needs to be treated securely and safely, with “zero trust”
00:28:45 – Are there any gaps in log data collection?
00:32:15 – Calyptia supports Lua rules
00:33:30 – How is collected data visualised?
00:39:00 – Wrap Up

Podcast: Play in new window | Download

Podcast (hybridcloud): Play in new window | Download