AIOPS
Written by: Marcel Koert B.S.E.E. | Posted on: | Category:

Introduction

In two of my previous blogs I talked about what AIOPS can do for you. Now I would like to talk to you about what AIOPS tooling needs to have to be as useful as possible.

Requirements for the AIOPS tools

• Ingesting data from multiple sources. • Analytics on real time data on the moment of ingestion. • Historical analysis of stored data. • Provide access to the data. • Storing the acquired data. • Use machine learning to analyse the data. • Being able to take action on result of analysis. • Large set of integration options.

Questions & Considerations

AIOPS

Ingesting data from multiple sources.

You will need to look at your data creating streams. Where is your data located and can the tool ingest this data from all those resources? Do not just think of logging ...

rate limiting
Written by: Marcel Koert B.S.E.E. | Posted on: | Category:

Rate Limiting & Throttling

Definitions

police man rate limiting speed
Rate Limiting:

Controlling the incoming and outgoing messages from your application so that they do not cross over a predetermined limit.

Throttling:

Controlling the incoming and outgoing messages on bases of resource measurements. These can be CPU/Disk IO/Network IO. But also, Threads used or functional measurements.

Why

Rate limiting & Throttling are used to protect you application from none agreed high loads. But you not only protect your application. You also protect your back ends because they get the load you send to them. The load to the back ends is not always a 1 to 1. The load can be a 1:N relation so for every call you ...

Written by: Marcel Koert B.S.E.E. | Posted on: | Category:

Quick over view.

All Applications that you write should have good logging. But what is good logging? Let’s start with a few No Brainers. Your applications uptime is always more important than your logging. If the data that you want to write is more important than it should not be in a log but in a database of some sort. Logging is only transient data.

We can define 3 types of logging:

  • Application log: All application failures, start , stop's with their reasons. (See log levels)
  • Audit logs: All functional actions done on behave of consumers, with the following data who, when and what.
  • Access logs: HTTP based log what ip connected to what end point at what time and with what code was answered.
Love logs

LogTypes


Audit


Why ...

Decoupled Application Monitoring
Written by: Marcel Koert B.S.E.E. | Posted on: | Category:

What are we doing now


Dashboard Monitoring

There is are a lot of new monitoring tools out there. The tools are becoming more sophisticated and there are more of them. How do you keep up with the new Application Monitoring tools when every time you want to try a new monitoring tool you have to update your applications? This slows down your speed of change. So why not decouple your applications from your monitoring tools.

It used to be that a lot of tooling is API based. This means you from your application send data to a API of your monitoring application. To do this your application has to talk the protocol that the Monitoring tool understands. And if you want to switch monitoring tools you will have to upgrade your application.

There ...

Written by: Marcel Koert B.S.E.E. | Posted on: | Category:
AIOPS and big data

Our applications and systems are generating more and more data. This is not just user generated data but also data generated by the Applications and Systems themselves. We are talking about logging from applications, systems, networks., etc., as well as metric from applications, systems, and networks. A small application can quickly generate 15 Megs of data-a-day, on a conservative side. (For comparison THE BIBLE is about 3.2 Megs). We as humans can’t read or comprehend this amount of data so when something goes wrong, most will search in the amount of data that they have to find out what is wrong.

Most people learn some of the errors by heart and search for those (90% of all data that is created is not used, ‘generally’ because we always look for ...

© 2019 Marcel Koert for MeloMar IT BV