====== Merge - incremental recalculation ======

{{tag> topic idm-acc merge controlled incremental dirty_state provisioning }}

@since 15.15.x

This page describes the **incremental recalculation** of MERGE controlled values introduced by the rework of provisioning merge cleanup. It complements the [[devel:documentation:systems:dev:system-mapping#merge_merge|Merge]] page , which still describes the conceptual model of controlled values; this page focuses on the **how** of the recalculation pipeline and its operational concerns.

<note important>The legacy ''evictControlledValuesCache'' flag and the synchronized full recalculation are kept as a **transitional fallback** only. The default for new attributes is now ''false'' and the production cleanup runs incrementally via dirty states.</note>

===== Motivation =====

Before //15.15.x// the recalculation worked like this:

  * The whole pipeline was guarded by a JVM-level ''synchronized'' block in ''DefaultSysSystemAttributeMappingService.recalculateAttributeControlledValues''.
  * The inner ''recalculateAttributeControlledValuesInternal'' ran in ''REQUIRES_NEW'' / ''SERIALIZABLE'' and recomputed the full controlled set for the whole mapping.
  * Triggered by an "evicted cache" flag on each ''SysSystemAttributeMapping''. Any save of a role-system-attribute set the flag; the next provisioning ran the full recalc.

In a real installation with ~14 000 roles this regularly **blocked all provisioning for tens of minutes** after a small bulk role change (e.g. import of 5 roles), because the lock and the SERIALIZABLE transaction serialized the whole tenant.

The new design replaces "full recompute on a global lock" with "**process only what changed**, lock-free":

  * Each save of a merge contributor enqueues a **single dirty state** for that role-system-attribute.
  * A dedicated link table tracks which role-system-attribute contributes which active controlled value, so deletes and changes can be reconciled locally instead of recomputing the full set.
  * Pending dirty states are drained either **lazily** (during the first provisioning that needs controlled values for the mapping) or **in bulk** by the recalculation LRT. No JVM lock, no SERIALIZABLE transaction.

===== Architecture =====

==== Key artifacts ====

  * **''IdmEntityState''** with code ''DIRTY_STATE''
      * Owner: ''SysRoleSystemAttribute'' (the changed role override).
      * Super owner id: parent ''SysSystemAttributeMapping'' id - lets the lazy drain filter pending states per mapping with a single DB query.
      * State: ''BLOCKED'' (queued for processing) or ''EXCEPTION'' (processing failed, kept for audit).
      * **Does not carry the computed value** - it is re-evaluated at drain time so multiple saves dedup to a single state and always reflect the latest definition.
  * **''sys_role_sys_attr_contr_val''** link table (entity ''SysRoleSystemAttributeControlledValue'')
      * Links each role-system-attribute to the active controlled value it currently contributes.
      * Has a DB ''UNIQUE'' constraint on ''(role_system_attribute_id, controlled_value_id)'' - lock-free race protection via ''DataIntegrityViolationException''.
      * Not audited; service is intentionally non-eventable (no Save/Delete processors needed).
  * **''AttributeControlledValuesInitTaskExecutor''** - one-time LRT that populates the link table for an existing installation (see [[#one-time_initialization|One-time initialization]]).
  * **''AttributeControlledValuesRecalculationTaskExecutor''** - the existing recalc LRT, kept but repurposed. With ''onlyEvicted=true'' it now drives the incremental drain per attribute mapping; with ''onlyEvicted=false'' it still runs the legacy synchronous full recompute (manual repair tool).

==== Dirty state lifecycle ====

''RoleSystemAttributeDirtyStateProcessor'' (CzechIdM event processor for ''SysRoleSystemAttribute'' CREATE/UPDATE) enqueues a dirty state when:

  * CREATE: the new role-system-attribute is a valid merge contributor (strategy = MERGE, not disabled, transform script is present).
  * UPDATE: any of these fields changed - ''strategyType'', ''disabledAttribute'', ''transformScript'', ''systemAttributeMapping'' - **and** the old or the new state is a valid merge contributor.

If a previous dirty state already exists for the same role-system-attribute, it is reused (deduplicated by owner). Repeated saves therefore never accumulate states.

DELETE does **not** enqueue a dirty state. The deletion cleanup is synchronous in ''RoleSystemAttributeDeleteProcessor'' (removes link records and runs orphan check on each released controlled value).

==== Drain ====

The drain (''DefaultSysRoleSystemAttributeControlledValueService.processIncremental(UUID mappingId)'') pages pending dirty states with a single filter ''(ownerType, superOwnerId=mappingId, code=DIRTY_STATE, state=BLOCKED)'' and processes them until the queue is empty:

  * **Add branch** - the role-system-attribute is still a valid contributor. The script is evaluated; the matching active controlled value is found (or created); the link record is upserted via ''ensureValue''; any historic duplicate of the same value is removed (invariant: a value on a mapping is either active or historic, never both).
  * **Remove branch** - the role-system-attribute is no longer a valid contributor (disabled / non-merge / empty script). All link records for this role-system-attribute are deleted; each released active controlled value runs through ''removeIfOrphan'' (see below).
  * **Defensive branches** - missing parameter, role-system-attribute already deleted, script exception, null or non-serializable value. These either silently drop the state or mark it ''EXCEPTION'' (kept for audit and admin investigation).

===== Active vs. historic invariant =====

A controlled value on a parent mapping is **either active or historic, never both**. The drain enforces this in two places:

  * In the **add branch**, after upserting an active value: any historic record for the same value/mapping is removed. This handles "re-enable" scenarios where a value previously moved to historic becomes active again.
  * In ''removeIfOrphan'', when the released value **still has contributors** (count of link records > 0): any historic duplicate is removed. This handles "delete one of several roles sharing the same value" - the value stays active, no historic is needed.
  * In ''removeIfOrphan'', when the released value becomes **orphan** (count = 0): the value is preserved as historic (''addHistoricValue'' before delete) and the active record is deleted. This is what provisioning needs as the "remove this from the resource" signal.

<note tip>This is the same semantic as the legacy ''setControlledValues'' reconcile, only enforced incrementally per role-system-attribute change instead of via a global recompute.</note>

===== Lazy drain during provisioning =====

''DefaultSysSystemAttributeMappingService.getCachedControlledAndHistoricAttributeValues'' calls ''processIncremental(attributeMapping.getId())'' **before** reading controlled values from the DB. The first provisioning after a role change therefore picks up the pending dirty states automatically; no separate task or admin action is needed.

The legacy ''evict'' block is still present in the same method as a transitional fallback for data with ''evictControlledValuesCache=true''. Because the default is now ''false'' and nothing sets it back to ''true'', the legacy block stays dormant for new installations.

===== One-time initialization =====

After upgrade to 15.15.x the ''sys_role_sys_attr_contr_val'' table is empty and has to be filled from existing role-system-attributes. This is done by **''AttributeControlledValuesInitTaskExecutor''**, started automatically by ''AccInitMergeControlledValueProcessor'' on application startup.

The task is executed **synchronously** via ''LongRunningTaskManager.executeSync'' - the application is fully ready only after init completes. This guarantees that the first provisioning call after upgrade sees a consistent state and does not race with the init.

The task iterates every ''SysRoleSystemAttribute'', evaluates the transform script and either:

  * inserts the link record (''SysRoleSystemAttributeControlledValue'') for the matching active controlled value, or
  * if no active controlled value matches yet (data drift), enqueues a dirty state so the next drain handles it.

==== Configuration ====

In the application profile (''application.properties''):

<code properties>
# One-time initialization of the merge controlled values table (sys_role_sys_attr_contr_val).
# Runs synchronously at application startup, blocks the start until finished
# (necessary so any first provisioning sees a complete state).
# The init long running task fills the SysRoleSystemAttributeControlledValue records for all existing role-system-attributes
# and sets this property to 'false' once it finishes successfully.
# - true: init will run on the next start (default for a fresh deploy)
# - false: init was already executed (set automatically after the first successful run)
# Set this back to 'true' manually only if the table was wiped or you need a full re-init.
# @since 15.15.x
idm.sec.acc.provisioning.controlledValues.init.enabled=true
</code>

<note tip>For ~10 000 role-system-attributes the initialization takes around 20 minutes on a typical machine. It runs in a single transaction, so a failure rolls back the partial state and the task is simply re-run on the next start.</note>

<note important>Until the init LRT finishes, no provisioning happens (synchronous startup). For very large installations plan the upgrade window accordingly. After successful init the property flips to ''false'' and the application starts normally on every next restart.</note>

===== Manual recalculation =====

Admins can still trigger ''AttributeControlledValuesRecalculationTaskExecutor'' from **Scheduler -> Long running tasks**. Form parameters:

  * ''system-uuid'' (mandatory)
  * ''entity-type'' (default ''IDENTITY'')
  * ''mapping-uuid'' (mandatory - the parent system mapping)
  * ''only-evicted'' (default ''true'') - **new semantic**: with ''true'' the task drives the incremental drain (''processIncremental'' per attribute mapping); with ''false'' it falls back to the legacy synchronous recompute (''recalculateAttributeControlledValues'') as a manual repair tool.

<note tip>The parameter name ''only-evicted'' is kept for backward compatibility. With the deprecation of the evict flag it now reads as "incremental mode" rather than "only attributes with evict=true".</note>

==== Common situations ====

  * **A merge value is missing on the target system after a role change.** Check pending dirty states for the mapping. If any are ''BLOCKED'' the next provisioning will drain them; if any are ''EXCEPTION'' the script failed - inspect the state body, fix the script and save the role-system-attribute again (creates a new ''BLOCKED'' state, old ''EXCEPTION'' stays for audit).
  * **A removed role still contributes its value to the resource.** Verify that the historic record exists in ''sys_attribute_contr_value'' for that value. If not, run the recalculation LRT manually for the mapping; if it does, force an update provisioning on an affected identity - the provisioning loop will send the remove instruction.
  * **The init LRT did not finish (application stuck on startup).** The task log is in **Scheduler -> Long running tasks**; only an ''EXCEPTION'' from a transform script breaks it. Fix the script and restart - the config flag is still ''true'', the task re-runs from scratch.

===== Related pages =====

  * [[devel:documentation:systems:dev:system-mapping#merge_merge|Merge]] - conceptual description of MERGE strategy and controlled values 
  * [[devel:documentation:application_configuration|Backend application properties]] - full configuration reference