# AeroTracker

Tracker for aeroespatial objects.

## Tracker
To run these services is mandatory to create some env variables like
"SATELLITE_NAME" and "SATELLITE_ID" for each satellite. 

To successfully run tracker service:

> $ cd src/services/tracker \
> $ sudo docker build . -t tracker_root \
> $ docker run --network=host tracker_root

#### Code explanation:
The main file is **"satellite_tracker.py"**, where a while loop is executed and every 3 seconds it
sends a request to an external API that provides information of a certain satellite. Then, some 
operations for providing even more data are performed and finally everything is sent to a RabbitMQ
queue.

## Monitor
For monitoring metrics related to cpu, memory and network a metricbeat module will be used.
To succesfully run:
> $ cd src/services/monitor \
> $ sudo docker build . -t monitor_root --network=host\
> $ sudo docker run -d --privileged --network=host monitor_root

#### Code explanation:
To monitor periodically all the values of a certain machine, a while loop will be running in 
**resources_monitor.py** main for sending data every second. For gathering data a method located
in the same file named as **"_get_all_resources_data"** will be used. It works by collecting
the outputs from other methods used for getting system information and finally making a dictionary.
Eventually, this final dictionary will be sent to elasticsearch.

# Modulo de calculo

## Configuration

The configuration of the calculus module is done in a json file such as config.json
This configuration file has three main sections:

### Fields

Here, the fields returned by elastic are mapped to names that will be used in the calculations later on.
The way to extract these is by adding a json object within the list which will represent a variable for
the calculations.

These json objects must have the name which will be used to reference the values and the path which will
be used to extract the data from Elastic's returned data.

> {
>     "name": "write_bytes_sda",
>     "path": ["_source", "disk_rw", "sda", "write_bytes"]
> },

in this example, it is expected that the values that will be addressed by the name "write_bytes_sda" are
indexed within the Elastic's returned data as follows:

> data["_source"]["dist_rw"]["sda"]["write_bytes"]

meaning that the data returned by elastic must be of the structure:

> {
>     _source: {
>         disk_rw: {
>             sda: {
>                 write_bytes : 1000
>             }
>         }
>     }
> }

### Calculus

This section is where the different calculations are defined. each entry of the list is a new calculation,
which will be named so that intermediate calculations can be used for further calculations. For example:

> {
>     "name": "delta_write_bytes_sda",
>     "function" : "delta",
>     "args" : ["write_bytes_sda"]
> },

As shown, the calculation "delta_write_bytes_sda" is the result of applying the function "delta" to the
values stored within the timeseries named "write_bytes_sda" which we declared earlier.

the args field contains a list of arguments which will be used in the call for the "delta" function. In this
case, the function only admits one, but for other functions, there could be more than one argument:

> {
>     "name": "heuristic",
>     "function" : "add",
>     "args" : ["heuristic_partial", "bat_heur"]
> }

In this last example, the "addition" function is used, which is a binary operator and thus requires 2 arguments.
The function module will have the functions declared and will declare the arity (number of arguments required)
of each function, so that creating a calculation with the wrong number of arguments will give an error and the
program will be able to tell the cause of the error.

The arguments can be either a reference (by name) to another data or an inmediate number. For example:

> {
>     "name": "cpu_time_idle_normalized",
>     "function" : "div",
>     "args" : ["delta_cpu_time_idle", 1000]
> },

The functions available are declared in the functions.py module.

Furthermore, the result of a calculation can be used as an argument for another calculation.

Instead of using functions for simple calculations, there is an option to express more complex calculations that mix
several different operations using the __expr__ object as follows:


> {
>     "name": "heuristic_test",
>     "expr": "add(mul($battery, 0.01, 0.25), mul(div($delta_cpu_time_idle, 1000), 0.5), mul(div($virtual_mem_available, $virtual_mem_total), 0.25))"
> },

in this example, there is a calculation named "heuristic_test" which is computed following the expression below, in which
additions, multiplications and substractions are used with both referenced variables and inmediate data. The expression is
parsed when reading the config and evaluated only when the calculation needs to be performed.

These expressions can only use the functions that are available for the __function__ objects as well and has the following syntax:

- function calls are allways prefix, including additions, multiplications, etc. this means that instead of the usual (a + b) notation,
the notation would actually be like +(a, b). note that, __+__ is not a function, __add__ is the name of the function to be used instead,
but it was choosen for ilustration purposes
- parenthesis are used to determine the list of arguments of a function
- commas separate the different arguments of a function
- when a reference is used as an argument in a function call, it must be preceded with $ to denote that the following name is a reference
- references cannot be functions and must be referencing an already existing field or calculation.


### Plot

This section is where the different ways of plotting are defined. each entry of the list is a new plot.
Each plot has a title and the fields that will be represented in the y and x axis. There is also a type
field which will contain the name of the type of graph that the user wants to plot. For example, lines or
bars, etc.

For the x axis, only one field can be represented, but in the y axis, a list of values can be plotted against
the x axis:

> {
>     "title": "heuristic",
>     "type": "lines",
>     "x": "timestamp",
>     "y": ["heuristic"]
> }


#### Technical debt

- In the calculus module, some functions do not return a timeseries but an scalar number. This could lead
to problems if an scalar is used when a timeseries is expected as an argument. Thus, in order to check for
this kinds of errors before executing the calculations, a "type" could be added to both the fields and the
calculus json objects. If that is done, in the declaration of functions inside functions.py, one could
declare the expected types of arguments for each function and thus, when datus.py is reading the configuration,
it could be able to tell whether a calculation is ill-formed because one of its arguments expected a timeseries
but was given a scalar number.

## datus

datus.py is a file whose objective is to download the data stored in Elastic in order to use that data to perform
the calculations specified in the configuration json file and then plotting them or sending them to other components.

This module uses a class Datus. each object of the class Datus require the filename of the configuration that they
will use and each has a dictionary (named calculus) which maps the names of fields and calculations specified in the
configuration into the result of downloading the Elastic data and the results of the calculations. Meaning that, when
we said in the configuration that a name was a reference to some value, the datus object would lookup this name inside
its dictionary and retrieve the value mapped.

The first thing that a datus object would want to do is to call the get_fields method, which will inside make a query
for Elastic and retrieve all the Elastic data specified in the configuration. it will also allocate memory for the
expected entries in the calculus dictionary. note that, since Elastic can return a lot of data in multiple messages,
This method could take a while to execute.

When allocating the memory, the datus object will perform some checks to see if the configuration is correct. These are:
- if each name in the configuration in fact references something (avoiding typos of names or references to unexisting
data for example)
- if a new name is not already taken by some other field or calculation, because this model does not support mutation of
variables
- if the function used inside a calculation has as many arguments as needed.
For each of these errors, an output message is printed so that the user knows what went wrong and the execution is aborted.

After that, there is the apply_functions method, which will perform the operations on the data.

finally, the plot_all method will call the graphics module in order to plot the data following what is specified in the
configuration.

There are some other methods, such as clear and load_config, which are there for a datus object to be able to reset if needed.

### Technical debt

- The datus does not have unit tests yet.
- The Datus class should have a method for reading the configuration and performing some checks for the correction of it,
instead of doing so while allocating data.
- The Datus class right now has to download all the data from elastic and then perform the calculations. Since this download
is done in several queries, it could be parallelized. Or maybe the calculations could be performed in chunks instead of waiting
to have all data downloaded before performing any calculation.

## expr

expr.py is a file to abstract the parsing, validating and evaluation of the expressions used in the config.
This is done using a class Expr whose objects represent expressions that know how to self evaluate with a given
dictionary to resolve references (which would be the dictionary that we have on datus)

The intended usage of this class would be by using the **parse_str** static method, which takes a string as an argument, which
must be the expression to be parsed. if the expression is malformed, for example, because it is missing a parenthesis,
then an exception would be raised because the parsing would fail. If the parsing succeeds, then an expression would be returned.

An expression is just a pair of a function and a list of arguments. These arguments can either be an inmediate value (a number),
a reference (which is a string with the name of the variable to lookup when evaluating) and other expressions. Thus, an expression
can contain subexpressions. For example:

> add(7, mul($foo, 0.5))

here, we have an expression that has the function add, and whose arguments are 7 and a subexpression, which has the function mul
and arguments $foo (a reference to a variable named foo) and 0.5.

Once a valid expr object is constructed with the **parse_str** static method, the methods validate and eval can be invoked.

Validate checks that the expression (along with all its subexpressions) are valid. This means that their arguments are either
an inmediate number, a valid reference (a reference that exists within a dictionary) or a valid subexpression.

Eval calls the function of the expression on the arguments. if an argument is another expression, this subexpression would be
evaluated. The intermediate results are stored in the "\_\_aux\_\_" key on the dictionary (which is needed to resolve the
references)

### Technical debt

- The expr does not have unit tests yet.
- The parsing of the expr could be optimized.
