Merge - incremental recalculation

@since 15.15.x

This page describes the incremental recalculation of MERGE controlled values introduced by the rework of provisioning merge cleanup. It complements the Merge page , which still describes the conceptual model of controlled values; this page focuses on the how of the recalculation pipeline and its operational concerns.

Before 15.15.x the recalculation worked like this:

• The whole pipeline was guarded by a JVM-level synchronized block in DefaultSysSystemAttributeMappingService.recalculateAttributeControlledValues.
• The inner recalculateAttributeControlledValuesInternal ran in REQUIRES_NEW / SERIALIZABLE and recomputed the full controlled set for the whole mapping.
• Triggered by an "evicted cache" flag on each SysSystemAttributeMapping. Any save of a role-system-attribute set the flag; the next provisioning ran the full recalc.

In a real installation with ~14 000 roles this regularly blocked all provisioning for tens of minutes after a small bulk role change (e.g. import of 5 roles), because the lock and the SERIALIZABLE transaction serialized the whole tenant.

The new design replaces "full recompute on a global lock" with "process only what changed, lock-free":

• Each save of a merge contributor enqueues a single dirty state for that role-system-attribute.
• A dedicated link table tracks which role-system-attribute contributes which active controlled value, so deletes and changes can be reconciled locally instead of recomputing the full set.
• Pending dirty states are drained either lazily (during the first provisioning that needs controlled values for the mapping) or in bulk by the recalculation LRT. No JVM lock, no SERIALIZABLE transaction.

• IdmEntityState with code DIRTY_STATE
  • Owner: SysRoleSystemAttribute (the changed role override).
  • Super owner id: parent SysSystemAttributeMapping id - lets the lazy drain filter pending states per mapping with a single DB query.
  • State: BLOCKED (queued for processing) or EXCEPTION (processing failed, kept for audit).
  • Does not carry the computed value - it is re-evaluated at drain time so multiple saves dedup to a single state and always reflect the latest definition.
• sys_role_sys_attr_contr_val link table (entity SysRoleSystemAttributeControlledValue)
  • Links each role-system-attribute to the active controlled value it currently contributes.
  • Has a DB UNIQUE constraint on (role_system_attribute_id, controlled_value_id) - lock-free race protection via DataIntegrityViolationException.
  • Not audited; service is intentionally non-eventable (no Save/Delete processors needed).
• AttributeControlledValuesInitTaskExecutor - one-time LRT that populates the link table for an existing installation (see One-time initialization).
• AttributeControlledValuesRecalculationTaskExecutor - the existing recalc LRT, kept but repurposed. With onlyEvicted=true it now drives the incremental drain per attribute mapping; with onlyEvicted=false it still runs the legacy synchronous full recompute (manual repair tool).

RoleSystemAttributeDirtyStateProcessor (CzechIdM event processor for SysRoleSystemAttribute CREATE/UPDATE) enqueues a dirty state when:

• CREATE: the new role-system-attribute is a valid merge contributor (strategy = MERGE, not disabled, transform script is present).
• UPDATE: any of these fields changed - strategyType, disabledAttribute, transformScript, systemAttributeMapping - and the old or the new state is a valid merge contributor.

If a previous dirty state already exists for the same role-system-attribute, it is reused (deduplicated by owner). Repeated saves therefore never accumulate states.

DELETE does not enqueue a dirty state. The deletion cleanup is synchronous in RoleSystemAttributeDeleteProcessor (removes link records and runs orphan check on each released controlled value).

The drain (DefaultSysRoleSystemAttributeControlledValueService.processIncremental(UUID mappingId)) pages pending dirty states with a single filter (ownerType, superOwnerId=mappingId, code=DIRTY_STATE, state=BLOCKED) and processes them until the queue is empty:

• Add branch - the role-system-attribute is still a valid contributor. The script is evaluated; the matching active controlled value is found (or created); the link record is upserted via ensureValue; any historic duplicate of the same value is removed (invariant: a value on a mapping is either active or historic, never both).
• Remove branch - the role-system-attribute is no longer a valid contributor (disabled / non-merge / empty script). All link records for this role-system-attribute are deleted; each released active controlled value runs through removeIfOrphan (see below).
• Defensive branches - missing parameter, role-system-attribute already deleted, script exception, null or non-serializable value. These either silently drop the state or mark it EXCEPTION (kept for audit and admin investigation).

A controlled value on a parent mapping is either active or historic, never both. The drain enforces this in two places:

• In the add branch, after upserting an active value: any historic record for the same value/mapping is removed. This handles "re-enable" scenarios where a value previously moved to historic becomes active again.
• In removeIfOrphan, when the released value still has contributors (count of link records > 0): any historic duplicate is removed. This handles "delete one of several roles sharing the same value" - the value stays active, no historic is needed.
• In removeIfOrphan, when the released value becomes orphan (count = 0): the value is preserved as historic (addHistoricValue before delete) and the active record is deleted. This is what provisioning needs as the "remove this from the resource" signal.

DefaultSysSystemAttributeMappingService.getCachedControlledAndHistoricAttributeValues calls processIncremental(attributeMapping.getId()) before reading controlled values from the DB. The first provisioning after a role change therefore picks up the pending dirty states automatically; no separate task or admin action is needed.

The legacy evict block is still present in the same method as a transitional fallback for data with evictControlledValuesCache=true. Because the default is now false and nothing sets it back to true, the legacy block stays dormant for new installations.

After upgrade to 15.15.x the sys_role_sys_attr_contr_val table is empty and has to be filled from existing role-system-attributes. This is done by AttributeControlledValuesInitTaskExecutor, started automatically by AccInitMergeControlledValueProcessor on application startup.

The task is executed synchronously via LongRunningTaskManager.executeSync - the application is fully ready only after init completes. This guarantees that the first provisioning call after upgrade sees a consistent state and does not race with the init.

The task iterates every SysRoleSystemAttribute, evaluates the transform script and either:

• inserts the link record (SysRoleSystemAttributeControlledValue) for the matching active controlled value, or
• if no active controlled value matches yet (data drift), enqueues a dirty state so the next drain handles it.

In the application profile (application.properties):

# One-time initialization of the merge controlled values table (sys_role_sys_attr_contr_val).
# Runs synchronously at application startup, blocks the start until finished
# (necessary so any first provisioning sees a complete state).
# The init long running task fills the SysRoleSystemAttributeControlledValue records for all existing role-system-attributes
# and sets this property to 'false' once it finishes successfully.
# - true: init will run on the next start (default for a fresh deploy)
# - false: init was already executed (set automatically after the first successful run)
# Set this back to 'true' manually only if the table was wiped or you need a full re-init.
# @since 15.15.x
idm.sec.acc.provisioning.controlledValues.init.enabled=true

Admins can still trigger AttributeControlledValuesRecalculationTaskExecutor from Scheduler → Long running tasks. Form parameters:

• system-uuid (mandatory)
• entity-type (default IDENTITY)
• mapping-uuid (mandatory - the parent system mapping)
• only-evicted (default true) - new semantic: with true the task drives the incremental drain (processIncremental per attribute mapping); with false it falls back to the legacy synchronous recompute (recalculateAttributeControlledValues) as a manual repair tool.

• A merge value is missing on the target system after a role change. Check pending dirty states for the mapping. If any are BLOCKED the next provisioning will drain them; if any are EXCEPTION the script failed - inspect the state body, fix the script and save the role-system-attribute again (creates a new BLOCKED state, old EXCEPTION stays for audit).
• A removed role still contributes its value to the resource. Verify that the historic record exists in sys_attribute_contr_value for that value. If not, run the recalculation LRT manually for the mapping; if it does, force an update provisioning on an affected identity - the provisioning loop will send the remove instruction.
• The init LRT did not finish (application stuck on startup). The task log is in Scheduler → Long running tasks; only an EXCEPTION from a transform script breaks it. Fix the script and restart - the config flag is still true, the task re-runs from scratch.

On the appliance environment the default synchronous startup initialization may be impractical for very large installations - the application is unavailable until the init LRT finishes (see One-time initialization above). For these cases the init can be disabled at startup and triggered later as a regular scheduled task. This trades a long single downtime for a shorter downtime plus a controlled operational window in which the new mechanism is not fully active.

1. Create an override properties file on the appliance, for example:
   /data/volumes/czechidm/application.properties.d/application-skip.properties
2. Disable the automatic init in that file:

   idm.sec.acc.provisioning.controlledValues.init.enabled=false

3. Restart the IdM service:

   systemctl restart iam-czechidm

4. Wait until IdM is fully up.
5. In Scheduler → Long running tasks, plan and execute the AttributeControlledValuesInitTaskExecutor task manually.

The concrete risks are:

• Delete or disable of a merge attribute during the window can wipe an active value from the target system - the empty link table makes removeIfOrphan treat shared values as orphans.
• Save of a merge attribute during the window persists into an incomplete link table; subsequent orphan checks on other roles may then misfire.
• No self-healing. With init.enabled=false the system never re-runs the init on its own. A forgotten or failed task leaves a partial state indefinitely.
• Init failure is atomic - one transaction, rollback to empty. Admin must fix the cause (typically a transform script) and re-trigger.
• The "no downtime + unsafe window" model is harder to communicate and audit than a clean downtime.

idm.sec.acc.provisioning.controlledValues.init.enabled=false

• Merge - conceptual description of MERGE strategy and controlled values
• Backend application properties - full configuration reference