# Scim and Migration Tooling

We need to be able to synchronise content from other directory or identity management systems. To do
this, we need the capability to have "pluggable" synchronisation drivers. This is because not all
deployments will be able to use our generic versions, or may have customisations they wish to
perform that are unique to them.

To achieve this we need a layer of separation - This effectively becomes an "extract, transform,
load" process. In addition this process must be _stateful_ where it can be run multiple times or
even continuously and it will bring kanidm into synchronisation.

We refer to a "synchronisation" as meaning a complete successful extract, transform and load cycle.

There are three expected methods of using the synchronisation tools for Kanidm

- Kanidm as a "read only" portal allowing access to its specific features and integrations. This is
  less of a migration, and more of a way to "feed" data into Kanidm without relying on its internal
  administration features.
- "Big Bang" migration. This is where all the data from another IDM is synchronised in a single
  execution and applications are swapped to Kanidm. This is rare in larger deployments, but may be
  used in smaller sites.
- Gradual migration. This is where data is synchronised to Kanidm and then both the existing IDM and
  Kanidm co-exist. Applications gradually migrate to Kanidm. At some point a "final" synchronisation
  is performed where Kanidm 'gains authority' over all identity data and the existing IDM is
  disabled.

In these processes there may be a need to "reset" the synchronised data. The diagram below shows the
possible work flows which account for the above.

                              ┏━━━━━━━━━━━━━━━━━┓
                              ┃                 ┃
                              ┃    Detached     ┃
    ┌──────────────────────┬──┃ (Initial State) ┃◀─────────────────────────┐
    │                      │  ┃                 ┃                          │
    │                      │  ┗━━━━━━━━━━━━━━━━━┛                          │
    │                      └──────────────────────────┐                    │
    │                                                 │                    │
    ├───────────────────────┬─────────────────────┐   │                    │
    │  ┌─────────────┐      │   ┌─────────────┐   │   │   ┌─────────────┐  │
    │  │             │      │   │             │───┘   │   │             │  │
    │  │   Initial   │      │   │   Active    │       │   │    Final    │  │
    └─▶│ Synchronise │──────┴──▶│ Synchronise │───────┴──▶│ Synchronise │──┤
       │             │          │             │           │             │  │
       └─────────────┘          └─────────────┘           └─────────────┘  │
              │                        │                                   │
              │                        │                  ┌─────────────┐  │
              │                        │                  │             │  │
              │                        │                  │    Purge    │  │
              └────────────────────────┴─────────────────▶│   Content   │──┘
                                                          │             │
                                                          └─────────────┘

Kanidm starts in a "detached" state from the extern IDM source.

For Kanidm as a "read only" application source the Initial synchronisation is performed followed by
periodic active (partial) synchronisations. At anytime a full initial synchronisation can re-occur
to reset the data of the provider. The provider can be reset and removed by a purge which reset's
Kanidm to a detached state.

For a gradual migration, this process is the same as the read only application. However when ready
to perform the final cut over a final synchronisation is performed, which retains the data of the
external system and grants Kanidm the authority over it. This then moves Kanidm back to the detached
state, but with a full cope of the provider data.

A "big bang" migration is this same process, but the "final" synchronisation is the first and only
step required, where all data is loaded and then immediately granted authority to Kanidm.

## ETL process

### Extract

First a user must be able to retrieve their data from their supplying IDM source. Initially we will
target LDAP and systems with LDAP interfaces, but in the future there is no barrier to supporting
other transports.

To achieve this, we initially provide synchronisation primitives in the
[ldap3 crate](https://github.com/kanidm/ldap3).

### Transform

This process will be custom developed by the user, or may have a generic driver that we provide. Our
generic tools may provide attribute mapping abilities so that we can allow some limited
customisation.

### Load

Finally to load the data into Kanidm, we will make a SCIM interface available. SCIM is a "spiritual
successor" to LDAP, and aligns with Kani's design. SCIM allows structured data to be uploaded
(unlike LDAP which is simply strings). Because of this SCIM will allow us to expose more complex
types that previously we have not been able to provide.

The largest benefit to SCIM's model is its ability to perform "batched" operations, which work with
Kanidm's transactional model to ensure that during load events, that content is always valid and
correct.

## Configuring a Synchronisation Provider in Kanidm

Kanidm has a strict transactional model with full ACID compliance. Attempting to create an external
model that needs to interoperate with Kanidm's model and ensure both are compliant is fraught with
danger. As a result, Kanidm sync providers _should_ be stateless, acting only as an ETL bridge.

Additionally syncproviders need permissions to access and write to content in Kanidm, so it also
necessitates Kanidm being aware of the sync relationship.

For this reason a syncprovider is a derivative of a service account, which also allows storage of
the _state_ of the synchronisation operation. An example of this is that LDAP syncrepl provides a
cookie defining the "state" of what has been "consumed up to" by the ETL bridge. During the load
phase the modified entries _and_ the cookie are persisted. This means that if the operation fails
the cookie also rolls back allowing a retry of the sync. If it succeeds the next sync knows that
kanidm is in the correct state. Graphically:

    ┌────────────┐                    ┌────────────┐                   ┌────────────┐
    │            │                    │            │     Retrieve      │            │
    │            │                    │            │──────Cookie──────▶│            │
    │            │                    │            │                   │            │
    │            │                    │            │    Provide        │            │
    │            │                    │            │◀────Cookie────────│            │
    │            │   Sync Request     │            │                   │            │
    │  External  │◀───With Cookie─────│    ETL     │                   │            │
    │    IDM     │                    │   Bridge   │                   │   Kanidm   │
    │            │   Sync Response    │            │                   │            │
    │            │────New Cookie─────▶│            │                   │            │
    │            │                    │            │                   │            │
    │            │                    │            │  Upload Entries   │            │
    │            │                    │            │──Persist Cookie──▶│            │
    │            │                    │            │                   │            │
    │            │                    │            │◀─────Result───────│            │
    └────────────┘                    └────────────┘                   └────────────┘

At any point the operation _may_ fail, so by locking the state with the upload of entries this
guarantees correct upload has succeeded and persisted. A success really means it!

## SCIM

### Authentication to the endpoint

This will be based on Kanidm's existing authentication infrastructure, allowing service accounts to
use bearer tokens. These tokens will internally bind that changes from the account MUST contain the
associated state identifier (cookie).

### Batch Operations

Per [RFC7644 section 3.7](https://datatracker.ietf.org/doc/html/rfc7644#section-3.7)

A requirement of the sync account will be a PATCH request to update the state identifier as the
first operation of the batch request. Failure to do so will result in an error.

### Schema and Attributes

SCIM defines a number of "generic" schemas for User's and Group's. Kanidm will provide its own
schema definitions that extend or replace these. TBD.

## Post Migration Concerns

### Reattaching a Provider Post Final Sync

In the case that a provider is re-attached after it has been through a final synchronisation,
entries that Kanidm now has authority over will NOT be synced and will be highlighted as conflicts.
The administrator then needs to decide how to proceed with these conflicts determining which data
source is the authority on the information.

## Internal Batch Update Operation Phases

We have to consider in our batch updates that there are multiple stages of the update. This is
because we need to consider that at any point the lifecycle of a presented entry may change within a
single batch. Because of this, we have to treat the operation differently within kanidm to ensure a
consistent outcome.

Additionally we have to "fail fast". This means that on any conflict the sync will abort and the
administrator must intervene.

To understand why we chose this, we have to look at what happens in a "soft fail" condition.

In this example we have an account named X and a group named Y. The group contains X as a member.

When we submit this for an initial sync, or after the account X is created, if we had a "soft" fail
during the import of the account, we would reject it from being added to Kanidm but would then
continue with the synchronisation. Then the group Y would be imported. Since the member pointing to
X would not be valid, it would be silently removed.

At this point we would have group Y imported, but it has no members and the account X would not have
been imported. The administrator may intervene and fix the account X to allow sync to proceed.
However this would not repair the missing group membership. To repair the group membership a change
to group Y would need to be triggered to also sync the group status.

Since the admin may not be aware of this, it would silently mean the membership is missing.

To avoid this, by "failing fast" if account X couldn't be imported for any reason, than we would
stop the whole sync process until it could be repaired. Then when repaired both the account X and
group Y would sync and the membership would be intact.

### Phase 1 - Validation of Update State

In this phase we need to assert that the batch operation can proceed and is consistent with the
expectations we have of the server's state.

Assert the token provided is valid, and contains the correct access requirements.

From this token, retrieve the related synchronisation entry.

Assert that the batch updates from and to state identifiers are consistent with the synchronisation
entry.

Retrieve the `sync_parent_uuid` from the sync entry.

Retrieve the `sync_authority` value from the sync entry.

### Phase 2 - Entry Location, Creation and Authority

In this phase we are ensuring that all the entries within the operation are within the control of
this sync domain. We also ensure that entries we intend to act upon exist with our authority markers
such that the subsequent operations are all "modifications" rather than mixed create/modify

For each entry in the sync request, if an entry with that uuid exists retrieve it.

- If an entry exists in the database, assert that its `sync_parent_uuid` is the same as our
  agreements.
  - If there is no `sync_parent_uuid` or the `sync_parent_uuid` does not match, reject the
    operation.

- If no entry exists in the database, create a "stub" entry with our `sync_parent_uuid`
  - Create the entry immediately, and then retrieve it.

### Phase 3 - Entry Assertion

Remove all attributes in the sync that are overlapped with our `sync_authority` value.

For all uuids in the entry present set Assert their attributes match what was synced in. Resolve
types that need resolving (name2uuid, externalid2uuid)

Write all

### Phase 4 - Entry Removal

For all uuids in the `delete_uuids` set: if their `sync_parent_uuid` matches ours, assert they are
deleted (recycled).

### Phase 5 - Commit

Write the updated "state" from the request `to_state` to our current state of the sync

Write an updated "authority" value to the agreement of what attributes we can change.

Commit the txn.
