# Logging

Logging is how the server communicates to developers and administrators about the state of the
service, and how operations are performing and what they are doing. It's important this is clear in
how it communicates.

## Use Cases

### Developer Bug Reports

A developer should be able to see the internal state of the server, and why any decision or logic
path was taken, so that errors or logic can be analysed post-incident. The information in the log
and the source code should be enough to resolve any issues, as we may not have LLDB access to any
consumers site, or any effective reproducer.

### Security Audits

We must be able to see why any security decision was made, such as credential validation, access
control application, or group/claim issuing to a session. This should be connected to the IP and
other identifiers of the caller.

### Error Analysis

For an administrator, they must be able to determine why an operation is failing in detail so they
can advise on how a consumer or user could change their behaviour to improve the situation (beyond
the error messages we return).

### Performance

Administrators and Developers should be able to analyse fine grained information about the
performance of any operation, and make informed decisions about tuning (such as caches or or
threads), and developers should be able to identify code paths that are under pressure and could be
targets for improvement.

### Containers/Systemd

Logs should be emitted on stdout/stderr as this is the easiest interface for existing log
aggregation systems to collect data from.

## Details

As developers we should indicate what messages are relevant to what use case as part of the message.
Log levels are used in other services, but that allows messages to be missed. Instead we always log
every "service", but filter them to different locations.

This leads to the following log categories:

- Analysis
  - Display of all logic branches and why decision points or paths taken
  - A unique event ID that associates related log messages
- Performance
  - Cache and DB metrics available
  - Performance frames of timing of key points
  - Structure of the performance frames to understand the execution paths taken.
  - Display of query optimisation
  - Display of query planning and application
- Failure (server failure)
  - Hard Errors
- Warning (admin should take action)
  - Possible misconfiguration
- OperationError (user mistake, op mistake etc)
  - All error reports and finalised result summaries logged
  - The unique event ID is provided in any operation success or failure.
- Security (aka audit)
  - Filtering of security sensitive attributes (via debug/display features)
  - Display of sufficient information to establish a security picture of connected actions via the
    user's uuid/session id.
  - Tracking of who-changed-what-when-why
- Replication
  - Both replication consumers and providers log when they make runs.
  - Errors in replication should surface as such.

It can be seen pretty quickly that multiple message types are useful across categories. For example,
the unique event id for all messages, how hard errors affect operation errors or how an operation
error can come from a security denial.

Logging must also remain a separate thread and async for performance.

This means that the best way to declare these logs is a unified log which can be filtered based on
the admins or consumers needs.

## API

For all types, it's important that we can associate all related events correctly. When the operation
initiates we assign an event-id that is part of the audit trail.

### Statistics

Stats should be accumulated in a statistics variable so that we can determine possible tuning and
other events related. Useful stats would be:

- Cache Hits
- Cache Misses
- Cache Inclusions

- Number of Searches
- Number of Entries Modified

This would be then logged as a structured line such as:

```json
{ "entry_cache_miss": 8, "idl_cache_miss": 8, "entry_cache_hit": 16', .... }
```

This would also then be fed back to the global stats thread for averaging.

### System Performance

The key metric for performance is time-in-function so it would be good to be able to build a value
like:

```json
{
    "name": "do_search",
    "time_ns": 130,
    "pct": 100,
    "called": [
        {
            "name": "filter2idl",
            "time_ns": 23',
            "called": [],
        },
        {
            ...
        }
    ]
}
```

This would allow a rich view of how much time went to any function at a high level, as then further
investigation can occur.

### SQL Query Analysis

To analyse a query we need:

- The original query
- The optimised version, with index tagging/profiling choices.
- The idl's that were loaded and how the query was applied
- The idl of the final result set.

### Security Events

- What access controls were considered?
- Who authenticated and where from?
- Audit of who modified what when why.

### Internal Analysis

This is generally what is "debug" logging, which is just decision points and verbose descriptions of
what we went where.

### Admin Notification

This is warnings or errors that the admin should be aware of.

### User Events

This must associate what happened for a user
