Merge - incremental recalculation

Merge - incremental recalculation

topic, idm-acc, merge, controlled, incremental, dirty state, provisioning

@since 15.15.x

This page describes the incremental recalculation of MERGE controlled values introduced by the rework of provisioning merge cleanup. It complements the Merge page , which still describes the conceptual model of controlled values; this page focuses on the how of the recalculation pipeline and its operational concerns.

The legacy evictControlledValuesCache flag and the synchronized full recalculation are kept as a transitional fallback only. The default for new attributes is now false and the production cleanup runs incrementally via dirty states.

Controlled values are stored as Java Serializable in a bytea column on SysAttributeControlledValue. All equality checks - active vs. historic matching, deduplication and the value-based filter - compare the serialized bytes. This is reliable for simple primitive types like String, Boolean and numeric types. For complex objects (Map, List of mixed entries, custom DTOs) the serialized form is not guaranteed to be stable across runs, JVM upgrades or even calls in the same process - two semantically equal values may serialize to different bytes and silently appear as separate records (active+historic split, missing dedup, broken value lookup). Always return a simple constant value from a merge transform script.

Motivation

Before 15.15.x the recalculation worked like this:

The whole pipeline was guarded by a JVM-level synchronized block in DefaultSysSystemAttributeMappingService.recalculateAttributeControlledValues.
The inner recalculateAttributeControlledValuesInternal ran in REQUIRES_NEW / SERIALIZABLE and recomputed the full controlled set for the whole mapping.
Triggered by an "evicted cache" flag on each SysSystemAttributeMapping. Any save of a role-system-attribute set the flag; the next provisioning ran the full recalc.

In a real installation with ~14 000 roles this regularly blocked all provisioning for tens of minutes after a small bulk role change (e.g. import of 5 roles), because the lock and the SERIALIZABLE transaction serialized the whole tenant.

The new design replaces "full recompute on a global lock" with "process only what changed, lock-free":

Each save of a merge contributor enqueues a single dirty state for that role-system-attribute.
A dedicated link table tracks which role-system-attribute contributes which active controlled value, so deletes and changes can be reconciled locally instead of recomputing the full set.
Pending dirty states are drained either lazily (during the first provisioning that needs controlled values for the mapping) or in bulk by the recalculation LRT. No JVM lock, no SERIALIZABLE transaction.

Architecture

Key artifacts

IdmEntityState with code DIRTY_STATE
- Owner: SysRoleSystemAttribute (the changed role override).
- Super owner id: parent SysSystemAttributeMapping id - lets the lazy drain filter pending states per mapping with a single DB query.
- State: BLOCKED (queued for processing) or EXCEPTION (processing failed, kept for audit).
- Does not carry the computed value - it is re-evaluated at drain time so multiple saves dedup to a single state and always reflect the latest definition.
sys_role_sys_attr_contr_val link table (entity SysRoleSystemAttributeControlledValue)
- Links each role-system-attribute to the active controlled value it currently contributes.
- Has a DB UNIQUE constraint on (role_system_attribute_id, controlled_value_id) - lock-free race protection via DataIntegrityViolationException.
- Not audited; service is intentionally non-eventable (no Save/Delete processors needed).
AttributeControlledValuesInitTaskExecutor - one-time LRT that populates the link table for an existing installation (see One-time initialization).
AttributeControlledValuesRecalculationTaskExecutor - the existing recalc LRT, kept but repurposed. With onlyEvicted=true it now drives the incremental drain per attribute mapping; with onlyEvicted=false it still runs the legacy synchronous full recompute (manual repair tool).

Dirty state lifecycle

RoleSystemAttributeDirtyStateProcessor (CzechIdM event processor for SysRoleSystemAttribute CREATE/UPDATE) enqueues a dirty state when:

CREATE: the new role-system-attribute is a valid merge contributor (strategy = MERGE, not disabled, transform script is present).
UPDATE: any of these fields changed - strategyType, disabledAttribute, transformScript, systemAttributeMapping - and the old or the new state is a valid merge contributor.

If a previous dirty state already exists for the same role-system-attribute, it is reused (deduplicated by owner). Repeated saves therefore never accumulate states.

DELETE does not enqueue a dirty state. The deletion cleanup is synchronous in RoleSystemAttributeDeleteProcessor (removes link records and runs orphan check on each released controlled value).

Drain

The drain (DefaultSysRoleSystemAttributeControlledValueService.processIncremental(UUID mappingId)) pages pending dirty states with a single filter (ownerType, superOwnerId=mappingId, code=DIRTY_STATE, state=BLOCKED) and processes them until the queue is empty:

Add branch - the role-system-attribute is still a valid contributor. The script is evaluated; the matching active controlled value is found (or created); the link record is upserted via ensureValue; any historic duplicate of the same value is removed (invariant: a value on a mapping is either active or historic, never both).
Remove branch - the role-system-attribute is no longer a valid contributor (disabled / non-merge / empty script). All link records for this role-system-attribute are deleted; each released active controlled value runs through removeIfOrphan (see below).
Defensive branches - missing parameter, role-system-attribute already deleted, script exception, null or non-serializable value. These either silently drop the state or mark it EXCEPTION (kept for audit and admin investigation).

Active vs. historic invariant

A controlled value on a parent mapping is either active or historic, never both. The drain enforces this in two places:

In the add branch, after upserting an active value: any historic record for the same value/mapping is removed. This handles "re-enable" scenarios where a value previously moved to historic becomes active again.
In removeIfOrphan, when the released value still has contributors (count of link records > 0): any historic duplicate is removed. This handles "delete one of several roles sharing the same value" - the value stays active, no historic is needed.
In removeIfOrphan, when the released value becomes orphan (count = 0): the value is preserved as historic (addHistoricValue before delete) and the active record is deleted. This is what provisioning needs as the "remove this from the resource" signal.

This is the same semantic as the legacy setControlledValues reconcile, only enforced incrementally per role-system-attribute change instead of via a global recompute.

Lazy drain during provisioning

DefaultSysSystemAttributeMappingService.getCachedControlledAndHistoricAttributeValues calls processIncremental(attributeMapping.getId()) before reading controlled values from the DB. The first provisioning after a role change therefore picks up the pending dirty states automatically; no separate task or admin action is needed.

The legacy evict block is still present in the same method as a transitional fallback for data with evictControlledValuesCache=true. Because the default is now false and nothing sets it back to true, the legacy block stays dormant for new installations.

One-time initialization

After upgrade to 15.15.x the sys_role_sys_attr_contr_val table is empty and has to be filled from existing role-system-attributes. This is done by AttributeControlledValuesInitTaskExecutor, started automatically by AccInitMergeControlledValueProcessor on application startup.

The task is executed synchronously via LongRunningTaskManager.executeSync - the application is fully ready only after init completes. This guarantees that the first provisioning call after upgrade sees a consistent state and does not race with the init.

The task iterates every SysRoleSystemAttribute, evaluates the transform script and either:

inserts the link record (SysRoleSystemAttributeControlledValue) for the matching active controlled value, or
if no active controlled value matches yet (data drift), enqueues a dirty state so the next drain handles it.

Configuration

In the application profile (application.properties):

# One-time initialization of the merge controlled values table (sys_role_sys_attr_contr_val).
# Runs synchronously at application startup, blocks the start until finished
# (necessary so any first provisioning sees a complete state).
# The init long running task fills the SysRoleSystemAttributeControlledValue records for all existing role-system-attributes
# and sets this property to 'false' once it finishes successfully.
# - true: init will run on the next start (default for a fresh deploy)
# - false: init was already executed (set automatically after the first successful run)
# Set this back to 'true' manually only if the table was wiped or you need a full re-init.
# @since 15.15.x
idm.sec.acc.provisioning.controlledValues.init.enabled=true

For ~10 000 role-system-attributes the initialization takes around 20 minutes on a typical machine. It runs in a single transaction, so a failure rolls back the partial state and the task is simply re-run on the next start.

Until the init LRT finishes, no provisioning happens (synchronous startup). For very large installations plan the upgrade window accordingly. After successful init the property flips to false and the application starts normally on every next restart.

Manual recalculation

Admins can still trigger AttributeControlledValuesRecalculationTaskExecutor from Scheduler → Long running tasks. Form parameters:

system-uuid (mandatory)
entity-type (default IDENTITY)
mapping-uuid (mandatory - the parent system mapping)
only-evicted (default true) - new semantic: with true the task drives the incremental drain (processIncremental per attribute mapping); with false it falls back to the legacy synchronous recompute (recalculateAttributeControlledValues) as a manual repair tool.

The parameter name only-evicted is kept for backward compatibility. With the deprecation of the evict flag it now reads as "incremental mode" rather than "only attributes with evict=true".

Common situations

A merge value is missing on the target system after a role change. Check pending dirty states for the mapping. If any are BLOCKED the next provisioning will drain them; if any are EXCEPTION the script failed - inspect the state body, fix the script and save the role-system-attribute again (creates a new BLOCKED state, old EXCEPTION stays for audit).
A removed role still contributes its value to the resource. Verify that the historic record exists in sys_attribute_contr_value for that value. If not, run the recalculation LRT manually for the mapping; if it does, force an update provisioning on an affected identity - the provisioning loop will send the remove instruction.
The init LRT did not finish (application stuck on startup). The task log is in Scheduler → Long running tasks; only an EXCEPTION from a transform script breaks it. Fix the script and restart - the config flag is still true, the task re-runs from scratch.

First time: Manual initialization on appliance

On the appliance environment the default synchronous startup initialization may be impractical for very large installations - the application is unavailable until the init LRT finishes (see One-time initialization above). For these cases the init can be disabled at startup and triggered later as a regular scheduled task. This trades a long single downtime for a shorter downtime plus a controlled operational window in which the new mechanism is not fully active.

Steps

Create an override properties file on the appliance, for example:
/data/volumes/czechidm/application.properties.d/application-skip.properties

Disable the automatic init in that file:

idm.sec.acc.provisioning.controlledValues.init.enabled=false

Restart the IdM service:
```
systemctl restart iam-czechidm
```
Wait until IdM is fully up.
In Scheduler → Long running tasks, plan and execute the AttributeControlledValuesInitTaskExecutor task manually.

Risks

This procedure leaves a window of inconsistent state between the application startup and the manual completion of the init task. During this window the new incremental mechanism does not have a complete picture of which role-system-attribute contributes which controlled value, so any change to a merge role-system-attribute may behave incorrectly. Treat merge role-system-attribute administration (save / disable / delete) as frozen until the manual init finishes.

The concrete risks are:

Delete or disable of a merge attribute during the window can wipe an active value from the target system - the empty link table makes removeIfOrphan treat shared values as orphans.
Save of a merge attribute during the window persists into an incomplete link table; subsequent orphan checks on other roles may then misfire.
No self-healing. With init.enabled=false the system never re-runs the init on its own. A forgotten or failed task leaves a partial state indefinitely.
Init failure is atomic - one transaction, rollback to empty. Admin must fix the cause (typically a transform script) and re-trigger.
The "no downtime + unsafe window" model is harder to communicate and audit than a clean downtime.

After AttributeControlledValuesInitTaskExecutor finishes successfully, persist the disabled flag through the IdM GUI (Nastavení → Konfigurace aplikace) so the override file can be safely removed - if the override file were deleted on its own, the next restart would re-trigger the synchronous init. Set the property via GUI:

idm.sec.acc.provisioning.controlledValues.init.enabled=false

and then delete the original override file from /data/volumes/czechidm/application.properties.d/.

Merge - conceptual description of MERGE strategy and controlled values
Backend application properties - full configuration reference

Table of Contents