Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Next revision Both sides next revision
devel:documentation:synchronization:dev:synchronization [2019/02/26 12:15]
kotisovam parts (intro about 2 modes of sync) moved to admin guide 		devel:documentation:synchronization:dev:synchronization [2020/10/09 09:04]
husniko [Entities that supports sync]
Line 1: Line 1:
-====== Two modes of synchronization ======+====== Synchronization (of identities) ======
 {{tag> sync identity}} {{tag> sync identity}}
  
 +The basic task of the synchronization is to ensure the correct state of the data on the end system and in IdM. Typically users´ accounts are considered as data. The correct state is defined both by the data in IdM (account management) and by the IdM configuration itself - usually by setting provisioning and synchronization on a given system.
 +
 +===== Entities that supports sync =====
 +
 +|**Name**                                                |**Entity name (DTO)**                         |**More details**  |
 +| Identity                  | IdmIdentityDto            |[[..:..:synchronization|]]                            |
 +| Contractual relationship                  | IdmIdentityContractDto            |[[..:..:synchronization:dev:relation-sync|]]                          |
 +| Time slices of contractual relationship                  | IdmContractSliceDto            |[[..:..:synchronization:dev:contract-slice-sync|]]                          |
 +| Tree                  | IdmTreeNodeDto            |[[..:..:synchronization:dev:tree-sync|]]                         |
 +| Role                  | IdmRoleDto            |[[..:..:synchronization:dev:role-sync|]]                          |
 +| Role catalogue                  | IdmRoleCatalogueDto            |[[..:..:synchronization:dev:role-catalogue-sync|]]                          |
 +| |
 + 
 +
 +<note tip>Apart from accounts, it is also possible to synchronize other types of entities (roles, trees, ...). However, to make things more simple, synchronization is considered only between an account and an identity in this text.</note>
 +
 +The usual synchronization process is as follows:
 +
 +  - **Finding the changed accounts** on the end system.
 +  - **Iteration of the changed accounts** and evaluation of the situation for each account (account in IdM).
 +  - **Action performance** for the situation found (e.g. creation of an identity in IdM).
 +  - **Running the subsequent operations** (e.g. provisioning updating an account on the end system).
 +
 +
 +===== Situation =====
 +During the synchronization process, the situation in which the account can be found based on the state in IdM is evaluated for every account.
 +Basic synchronization configuration means settings the type of action which should be done in a given situation.
 +
 +The synchronization situations are fixed and they are the following:
 +
 +==== Existent link ====
 +The situation when a corresponding account exists to a given account on the system ([[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/acc/src/main/java/eu/bcvsolutions/idm/acc/entity/AccAccount.java|AccAccount]]) in IdM.
 +
 +In this situation, it is possible to proceed to the following actions:
 +  * **Update the entity** - will update the identity linked to an account. The update is done based on mapping linked to the synchronization. At present, mapping enables to update the attributes of the **identity** itself, and also the attributes saved in the warehouse of **advanced attributes** as well as in **coded** warehouse. After saving an entity, standard provisioning will always be initiated.
 +
 +  * **Update account** - will initiate standard provisioning. Synchronization initiates an event, it does not do the provisioning itself.
 +
 +  * **Remove link** - will remove the link in IdM. meaning it will remove the account ([[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/acc/src/main/java/eu/bcvsolutions/idm/acc/entity/AccAccount.java|AccAccount]]) and the links between the identity and the account ([[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/acc/src/main/java/eu/bcvsolutions/idm/acc/entity/AccIdentityAccount.java|AccIdentityAccount]]). It does not modify the identity itself, does not initiate provisioning.
 +
 +  * **Remove link and related roles** - will remove the links like in the previous case but will also remove the identity roles which are linked to the corresponding ([[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/acc/src/main/java/eu/bcvsolutions/idm/acc/entity/AccIdentityAccount.java|AccIdentityAccount]]). In other words, it will remove the roles which assigned a given account to the identity.
 +
 +  * **Ignore** - This action does not perform any active operation.
 +
 +==== Non-existent link ====
 +Situation in which there is no link to a given account on the system (account in IdM), but an identity exists.
 +
 +Since the link does not exist, in this case the identity has been found through a correlation attribute. A correlation attribute can be any attribute from the related synchronization mapping (the correlation attribute is mandatory). 
 +
 +<note important>At present, the correlation attribute enables searching according to identity attributes (username, firstName, lastName, email, personal number) and searching according to extended (EAV) attributes.</note>
 +
 +For example, if you want to find (identify) identities in IdM based on the correspondence of the user name **username** and the account attribute **login**, you can use the following correlation attribute:
 + 
 + 
 +{{ :devel:documentation:synchronization:dev:attribute_mapping.png?nolink |}}
 +
 +----
 + 
 +In this situation, it is possible to proceed to the following actions:
 +
 +  * **Create link** - will create a link in IdM, meaning it will create an account ([[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/acc/src/main/java/eu/bcvsolutions/idm/acc/entity/AccAccount.java|AccAccount]]) and the link between the identity and the account ([[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/acc/src/main/java/eu/bcvsolutions/idm/acc/entity/AccIdentityAccount.java|AccIdentityAccount]]). It does not modify the identity itself, does not initiate provisioning.
 +
 +  * **Create link and update entity** (since 8.0) - The link is created in the same way as in the previous case. In addition, the entity is updated. The update is done based on mapping linked to the synchronization. After saving an entity, standard provisioning will always be initiated.
 +
 +  * **Create link and update account** - The link is created in the same way as in the previous case. In addition, the account is updated on the end system, meaning that the event for running the provisioning is initiated.
 +
 +  * **Ignore** - This action does not perform any active operation.
 +
 +==== Non-existent entity ====
 +Situation in which there is no identity in IdM to a given account on the system.
 +
 +In this situation, it is possible to proceed to the following actions:
 +
 +  * **Create entity** - will create an identity and a link in IdM. The creation will be done according to the mapping set up in synchronization. Creation of an entity will initiate provisioning (account update). 
 +
 +  * **Ignore** - This action does not perform any active operation.
 +
 +<note>In versions 7.6 - 8.1.x (in identity synchronization), the default contractual relationship (when creating a new identity) was not created! Since 8.2, it's controlled by specific settings of the synchronization.</note>
 +
 +==== Non-existent account ====
 +Situation in which there is no account on the end system to a given account in IdM.
 +
 +This situation may occur in case the connector supports the operation **DELETE**. This means that the connector is able to give information on which accounts have been deleted on the end system since the last synchronization. 
 +Yet this situation can be typically used in **reconciliation** when all the accounts in IdM are iterated overnight, verifying if an account exists on the end system - if it doesn´t a preset action is initiated.
 +
 +<note important>Although the response to the **DELETE** state has been implemented in synchronization, most connectors do not support this operation! The operation will not be available any more once you use **custom filter**</note>
 +
 +In this situation, it is possible to proceed to the following actions:
 +
 +  * **Create account** - Synchronization will only initiate the event for the linked entity [[https://github.com/bcvsolutions/CzechIdMng/blob/1a1ba4ff244967425e263c291965142e6612cbbd/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/event/IdentityEvent.java|IdentityEventType.UPDATE]] that will initiate provisioning and so the account will be created on the end system. 
 +
 +  * **Delete entity** - will delete the account in IdM, the link between the account and the identity, and the identity
 +
 +  * **Remove link** - will remove the link in IdM, meaning it will remove the account ([[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/acc/src/main/java/eu/bcvsolutions/idm/acc/entity/AccAccount.java|AccAccount]]) and the links between the identity and the account ([[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/acc/src/main/java/eu/bcvsolutions/idm/acc/entity/AccIdentityAccount.java|AccIdentityAccount]]). It does not modify the identity itself, does not initiate provisioning.
 +
 +  * **Remove link and related roles** - will remove the links like in the previous case but will also remove the identity roles which are linked to the corresponding ([[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/acc/src/main/java/eu/bcvsolutions/idm/acc/entity/AccIdentityAccount.java|AccIdentityAccount]]). In other words, it will remove the roles which assigned a given account to the identity.
 +
 +  * **Ignore** - This action does not perform any active operation.
 +
 +===== Connector synchronization vs. my own filter =====
 Synchronization supports two basic modes of searching the accounts suitable for synchronization. Synchronization supports two basic modes of searching the accounts suitable for synchronization.
 +The first mode is the use of synchronization mechanism which is provided directly by the connector. In such case, the method [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/ic/src/main/java/eu/bcvsolutions/idm/ic/service/api/IcConnectorFacade.java|IcConnectorFacade.synchronization]] is called on the connector after synchronization.
 +
 +The input of this method is the **token** defining where the synchronization should continue. In this case, the format of the token is defined directly by the connector (it can be a time mark, the order, ...) If the token is empty, then the synchronization will be launched for all the accounts of the system.
 +
 +The disadvantage of the synchronization in the connector is the impossibility to rule in details which of the accounts you want to synchronize (put together more complicated queries). On the contrary, the advantage is the possibility to use the operation **DELETE** . I. e. the situation in which the connector is able to report on its own which account has been deleted.
  
-  -   The first mode is the use of synchronization mechanism which is provided directly by the connector. In such case, the method [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/ic/src/main/java/eu/bcvsolutions/idm/ic/service/api/IcConnectorFacade.java|IcConnectorFacade.synchronization]] is called on the connector after synchronization.+<note important>In case of use of the internal synchronization in **Database Table Connector (connId)**, it is necessary to set up a column in its configuration, according to which the search will be made (**Change Log Column (Sync)**). However, if then you wish to synchronize using the custom filter (according to the same column), the filter will not work (nothing will be returned). In such case, it **necessary** to **remove** the settings from the configuration!</note> 
  
-  -   The second way is the **Custom filter**. This mode will compose the filter criterion first. This criterion is used to search account on the end system ([[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/ic/src/main/java/eu/bcvsolutions/idm/ic/service/api/IcConnectorFacade.java|IcConnectorFacade.search]]). The synchronization takes places above these results.  +The second way is the **Custom filter**. This mode will compose the filter criterion first. This criterion is used to search account on the end system ([[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/ic/src/main/java/eu/bcvsolutions/idm/ic/service/api/IcConnectorFacade.java|IcConnectorFacade.search]]). The synchronization takes places above these results.
  
 +This mode can be activated (**Use custom filter**) and set on the **Filter** folder.
 +The custom filter can be simply defined by choosing the attribute (**Filter by attribute**) by which you want to search, and the corresponding operation (**Filtering operation**).
  
-==== More complicated filter criterion ====+To save the final **token**, it is necessary to determine the account attribute which contains it. To do this, you may use the item **Token is contained in this attribute**.
  
 In case you need to create a more complicated filter criterion, it is possible to use this script. In case you need to create a more complicated filter criterion, it is possible to use this script.
Line 114: Line 221:
  
 ===== Specific synchronization options ===== ===== Specific synchronization options =====
 +{{tag>specific sync identity options settings}}
  
-{{ :tutorial:adm:synchronization_details_identities.png?nolink |}}+{{ :devel:documentation:synchronization:dev:sync-specific-settings.png? |}}
  
 You can configure additional synchronization options for specific uses: You can configure additional synchronization options for specific uses:
   * **Default role** - The value can be any role in CzechIdM. This value is used in the case that the synchronization links an existing system account to an existing or a new identity in CzechIdM. If the default role is specified, this role will be assigned to the identity for its main valid contractual relationship. Then the link to the account will be created with the property **Assigned by role** set to the default role. If the default role is empty, the link to the account will be created as well, only without the property "Assigned by role".   * **Default role** - The value can be any role in CzechIdM. This value is used in the case that the synchronization links an existing system account to an existing or a new identity in CzechIdM. If the default role is specified, this role will be assigned to the identity for its main valid contractual relationship. Then the link to the account will be created with the property **Assigned by role** set to the default role. If the default role is empty, the link to the account will be created as well, only without the property "Assigned by role".
 +  * **Assign default role to all valid contracts** - If the default role is selected and this option is checked, then the default role will not be assigned to the main contract, but to all valid or in future valid contracts of the identity. As in previous step, the assigning is realized via role-request and this request is executed immediately without approving.
     * The main use-case for this option is **initial linking of accounts during the reconciliation** of a system, where the accounts will be further managed by CzechIdM - e.g. LDAP, AD. The default role will be usually configured for provisioning on this system, see [[tutorial:adm:provisioning]].     * The main use-case for this option is **initial linking of accounts during the reconciliation** of a system, where the accounts will be further managed by CzechIdM - e.g. LDAP, AD. The default role will be usually configured for provisioning on this system, see [[tutorial:adm:provisioning]].
     * This option is supported in the following actions of the synchronization: Missing Entity -> Create entity, Not Linked -> Create link, Create link and update entity, Create link and update account.     * This option is supported in the following actions of the synchronization: Missing Entity -> Create entity, Not Linked -> Create link, Create link and update entity, Create link and update account.
Line 130: Line 239:
   * **After end, start the automatic role recalculation** - After synchronization correctly ended recalculation of automatic role will be started.    * **After end, start the automatic role recalculation** - After synchronization correctly ended recalculation of automatic role will be started. 
   * **Create default contracts for new identities** - If a new identity is created during synchronization, a default contract will be created for the identity. To use this feature, you must also enable creating default contracts in the [[devel:documentation:application_configuration:dev:backend#identity|application configuration]] (''idm.pub.core.identity.create.defaultContract.enabled=true'').   * **Create default contracts for new identities** - If a new identity is created during synchronization, a default contract will be created for the identity. To use this feature, you must also enable creating default contracts in the [[devel:documentation:application_configuration:dev:backend#identity|application configuration]] (''idm.pub.core.identity.create.defaultContract.enabled=true'').
 +
 +===== Differential sync (process only differences) =====
 +
 +The goal of **differential synchronization** is to update the synchronized entity only if at least one mapped attribute value has changed. This **prevents unnecessary event creation** in the system.
 +
 +**Differential synchronization only checks the values of the mapped attributes**. Ie. if the attribute that we do not map in IdM changes in the source account, the differential synchronization will not detect the change and the entity will not be saved.
 +
 +<note tip>You can turn on differential sync in both **reconciliation** and standard **sync** modes.</note>
 +
 +
 +If differential synchronization evaluates that the account has not changed against to the entity in IdM, the account is marked as **IGNORE** (blue color) in the sync log.
 +
 +**In the picture below**, only **one** change was detected.
 +
 +Therefore, **18 elements** are marked as **IGNORE** and **one** is marked as **SUCCESS** (green color):
 +
 +{{ :devel:documentation:synchronization:dev:diff_sync_use_.png?1000 |}}
 +
 +
 +{{ :devel:documentation:synchronization:dev:diff_sync_use_log.png?600 |}}
 +
 +==== How differential sync can be enabled? ====
 +
 +Differential sync can be enabled on the detail of sync, where **Process only differences** have to be checked.
 +
 +{{ :devel:documentation:synchronization:dev:diff_sync.png?1000 |}}
 +
 +==== When differential sync shoud be disabled ====
 +Differential synchronization can (especially in reconciliation mode) significantly **increase system speed**. **Normally there is no reason for it not to be activated**.
 +
 +The only situation where differential synchronization cannot be used is when we require all synchronized entities to be saved because of **project-specific processors** that are linked to the save event.
 +
 +
  
 ===== Events ===== ===== Events =====
Line 167: Line 309:
   * **Linked** - only if the action **Update entity** or **Update account** is configured and the property ''idm.sec.acc.provisioning.allowedAutoMappingOnExistingAccount'' is ''true''.   * **Linked** - only if the action **Update entity** or **Update account** is configured and the property ''idm.sec.acc.provisioning.allowedAutoMappingOnExistingAccount'' is ''true''.
     * Reason: The system entity is linked to an IdM entity. By removing "wish", we would effectively do auto mapping, because following provisioning attempts would be Update and not Create. So we can remove "wish" only if auto mapping is allowed.     * Reason: The system entity is linked to an IdM entity. By removing "wish", we would effectively do auto mapping, because following provisioning attempts would be Update and not Create. So we can remove "wish" only if auto mapping is allowed.
 +
 +===== User type (projection) =====
 +{{tag>projection user type}}
 +Since version **10.3.0**, identity synchronization supports sync of a user type:
 +
 +  * On the attribute detail, select the entity filed **User type (IdmFormProjectionDto)**.
 +  * The output of the **transformation from the system** can be a user-type **identifier** or its **code** (the **IdmFormProjectionDto** object).
 +
 +User type **provisioning** is also supported from this version:
 +
 +  * On the attribute detail, select the entity filed **User type (IdmFormProjectionDto)**.
 +  * Input value in **transformation to the system** is user type object (**IdmFormProjectionDto**).
 +
 +For provisioning a code of user type, you can use **this example:**
 +<code>
 +if (attributeValue != null) {
 +    return attributeValue.getCode();
 +}
 +</code>
 +
 +
 +
 +
  
  • by cem