Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
devel:documentation:adm:export_import [2020/04/02 15:20]
svandav [Future improvements] 		devel:documentation:adm:export_import [2021/09/17 15:25] (current)
apeterova formatting
Line 5: Line 5:
 This agenda is used to transfer configuration data from one IdM to another. A typical use scenario is when you already have IdM configured on a test environment and now you need to migrate the tested configuration to a production environment. This agenda is used to transfer configuration data from one IdM to another. A typical use scenario is when you already have IdM configured on a test environment and now you need to migrate the tested configuration to a production environment.
  
-Without this tool, you must manually transfer the settings. Ie. you must create roles, system definitions, including their configurations such as guarantors, linked attributes, etc. +Without this tool, you must manually transfer the settings. Ie. you must create roles, system definitions, including their configurations such as guarantors, linked attributes, etc. So the main goal is to facilitate configuration transfer between individual IdM systems.
-So the main goal is to facilitate configuration transfer between individual IdM systems.+
  
 This transmission is divided into two basic parts **Export** and **Import**: This transmission is divided into two basic parts **Export** and **Import**:
  
-{{ :devel:documentation:export_import_table.png?800 |}}+{{  ..:export_import_table.png?800  }}
  
 ===== Export ===== ===== Export =====
 +
 You can export data from the source IdM by using bulk operations over the appropriate agendas. You can export data from the source IdM by using bulk operations over the appropriate agendas.
  
-For example, if you want to export roles, then first select the appropriate roles in the role agenda and then run the **Export Roles** bulk operation. +For example, if you want to export roles, then first select the appropriate roles in the role agenda and then run the **Export Roles** bulk operation. The result of this operation will be to create a new export batch in the **Menu → Settings → Export/Import** agenda. This export batch will contain all exported data that can be downloaded in **ZIP** format.
-The result of this operation will be to create a new export batch in the **Menu -> Settings -> Export/Import** agenda. This export batch will contain all exported data that can be downloaded in **ZIP** format.+
  
-{{ :devel:documentation:export-roles.png?500 |}}+{{  ..:export-roles.png?500  }}
  
 ===== Import ===== ===== Import =====
-Import is performed in the target IdM, when in the same agenda **Menu -> Settings -> Export/Import**, we upload the batch that we exported in the source IdM (ZIP). 
  
 +Import is performed in the target IdM, when in the same agenda **Menu → Settings → Export/Import**, we upload the batch that we exported in the source IdM (ZIP).
  
 The import batch can then be run in the so-called **demo mode**. This mode serves to test what the given batch will actually change in the target IdM. The import batch can then be run in the so-called **demo mode**. This mode serves to test what the given batch will actually change in the target IdM.
Line 28: Line 27:
 When we are sure that we really want to make the changes that the batch makes, we can proceed to the launch of the import itself. When we are sure that we really want to make the changes that the batch makes, we can proceed to the launch of the import itself.
  
-{{ :devel:documentation:adm:import_log_tree.png?800 |}}+{{  .:import_log_tree.png?800  }}
  
 ===== Dry run mode ===== ===== Dry run mode =====
 +
 **Dry run** is a mode where we first want to check what changes will be made to the target IdM. For example, due to **authoritative mode**, some objects that are not part of the imported batch may be deleted during import. Starting dry run mode allows us to find out what **data will be deleted**. **Dry run** is a mode where we first want to check what changes will be made to the target IdM. For example, due to **authoritative mode**, some objects that are not part of the imported batch may be deleted during import. Starting dry run mode allows us to find out what **data will be deleted**.
  
Line 37: Line 37:
 <note>**Dry run mode** can be started using the blue play button on the export / import table.</note> <note>**Dry run mode** can be started using the blue play button on the export / import table.</note>
  
-{{ :devel:documentation:adm:dry-run-failed.png?800 |}}+{{  .:dry-run-failed.png?800  }}
  
 ===== How it works? ===== ===== How it works? =====
-Each export of a given type of object is implemented as a separate bulk action, which ensures correct data export. The export action must also ensure that the export metadata is correctly created. Export metadata is the file **export-batch.json**, which is part of each exported batch. This file contains metadata, such as the **batch name** and a descriptors for each exported object type. 
  
 +Each export of a given type of object is implemented as a separate bulk action, which ensures correct data export. The export action must also ensure that the export metadata is correctly created. Export metadata is the file **export-batch.json**, which is part of each exported batch. This file contains metadata, such as the **batch name** and a descriptors for each exported object type.
 ==== Export descriptors ==== ==== Export descriptors ====
-Export descriptors primarily define the order in which data is exported (the same order is used during import). In addition, each descriptor contains additional metadata that defines how the import should handle that type of object. The descriptor metadata also defines: 
  
-* **Supports authoritative mode** - Defines whether authoritative mode is enabled for this object type. If so, then the import removes other / redundant entities from the target IdM (for example, if the role contains additional guarantors that do not exist in the batch, they will be deleted).Parent fields are uses for find ID of super parents for this batch (system, role, ..). +Export descriptors primarily define the order in which data is exported (the same order is used during import). In addition, each descriptor contains additional metadata that defines how the import should handle that type of object. The descriptor metadata also defines:
-* **Parent fields** - Defines name of parent fields. This contains UUID of entity for which was DTO exported. This is important for authoritative import, because we need to delete others entity from target IdM. Parent fields are uses for find ID of super parents for this batch (system, role, ..). +
-* **Super parent filter property** - Defines name of filter for find all DTOs existing under same super parent on target system. Super parent DTO ia typically system or role. This field is mandatory for authoritative mode. This property is uses for find specific setter in Filter POJO. It means setter must exists in filter! +
-* **Advanced paring fields** -  Defines fields in DTO, where we need to use advanced paring strategy. It means that we need to check if UUID exists in target system. If not, we will use DTO from embedded map and try to find DTO by code. +
-* **Optional** - If is true and DTO will cannot be persisted, because some relation was not found, then only warning will be logged, but batch can continue.+
  
 +  * **Supports authoritative mode**  - Defines whether authoritative mode is enabled for this object type. If so, then the import removes other / redundant entities from the target IdM (for example, if the role contains additional guarantors that do not exist in the batch, they will be deleted).
 +  * **Parent fields**  - Defines name of parent fields. This contains UUID of entity for which was DTO exported. This is important for authoritative import, because we need to delete others entity from target IdM. Parent fields are uses for find ID of super parents for this batch (system, role, ..).
 +  * **Super parent filter property**  - Defines name of filter for find all DTOs existing under same super parent on target system. Super parent DTO ia typically system or role. This field is mandatory for authoritative mode. This property is uses for find specific setter in Filter POJO. It means setter must exists in filter!
 +  * **Advanced paring fields**  - Defines fields in DTO, where we need to use advanced paring strategy. It means that we need to check if UUID exists in target system. If not, we will use DTO from embedded map and try to find DTO by code.
 +  * **Optional**  - If is true and DTO will cannot be persisted, because some relation was not found, then only warning will be logged, but batch can continue.
 +  * **Excluded fields**  - Defines fields in DTO, which will be excluded during the import. It means this fields will be not changed on target IdM. If entity will not exists, then that fields will set to null. For example, the token in sync definition is excluded.
 +<note important>Since version 10.6.0 is **token**  field excluded from a import of system sync.</note>
  
 ===== What can be exported ===== ===== What can be exported =====
-Version **10.2.0** implements export of **roles**, **systems**, **application configurations**, and some related objects (such as form definitions or catalogs ...). 
  
-<note important>Passwords (stored in confidentila storage) are **never exported**. When you first import an object that contained a password (for example, a connector configuration (EAV attributes) or password in remote connector server), the password will not be filled in the target IdM. If it is an update of an existing object, then the password in the target IdM will **not be modified** in any way.</note>+Version **10.2.0**  implements export of **roles**, **systems**, **application configurations**, and some related objects (such as form definitions or catalogs …). 
 + 
 +<note important>Passwords (stored in confidential storage) are **never exported**. When you first import an object that contained a password (for example, a connector configuration (EAV attributes) or password in remote connector server), the password will not be filled in the target IdM. If it is an update of an existing object, then the password in the target IdM will **not be modified**  in any way.</note>
  
 ==== Roles ==== ==== Roles ====
Line 61: Line 64:
 Roles are exports with this related objects: Roles are exports with this related objects:
  
-* **More information (EAV)** (authoritative mode = **off**) +  * **More information (EAV)**  (authoritative mode = **off**) 
-* **Definition of assigned attributes** (authoritative mode = **on**) +  * **Definition of assigned attributes**  (authoritative mode = **on**) 
-* **Business roles** (authoritative mode = **on**) +  * **Business roles**  (authoritative mode = **on**) * **Incompatible roles**  (authoritative mode = **on**) 
-* **Incompatible roles** (authoritative mode = **on**) +  * **Role authorizer - defined by role**  (authoritative mode = **on**) 
-* **Role authorizer - defined by role** (authoritative mode = **on**) +  * **Role authorizer - defined by identity**  (authoritative mode = **on**) 
-* **Role authorizer - defined by identity** (authoritative mode = **on**) +  * **Role catalogue**  (authoritative mode = **on**). Besides a role catalog relation, the catalog definition itself is also exported. Only the catalog to which the role is linked is always exported and all catalogs defining the path to the root of the catalog tree. The catalogs themselves are not exported in authoritative mode (authoritative mode = **off**) 
-* **Role catalogue** (authoritative mode = **on**). Besides a role catalog relation, the catalog definition itself is also exported. Only the catalog to which the role is linked is always exported and all catalogs defining the path to the root of the catalog tree. The catalogs themselves are not exported in authoritative mode (authoritative mode = **off**) +  * **Permissions**  (authoritative mode = **on**) 
-* **Permissions** (authoritative mode = **on**) +  * **Automatic roles**  - Version **10.2.0**  doesn't implement export of automatic roles. 
-* **Automatic roles** - Version **10.2.0** doesn't implement export of automatic roles. +  * **Systems**  - Relation between role and system is not exported within role export. **This relations are exported within export of systems**.
-* **Systems** - Relation between role and system is not exported within role export. **This relations are exported within export of systems**. +
 ==== Systems ==== ==== Systems ====
 +
 <note tip>Systems may have a relation to a roles (provisioning breaks or roles that allocate the system), so you must always export and import roles before importing systems.</note> <note tip>Systems may have a relation to a roles (provisioning breaks or roles that allocate the system), so you must always export and import roles before importing systems.</note>
  
 Systems are exports with this related objects: Systems are exports with this related objects:
  
-* **Connector configuration** (authoritative mode = **off**) +  * **Connector configuration**  (authoritative mode = **off**) 
-* **Connector pooling configuration** (authoritative mode = **on**) +  * **Connector pooling configuration**  (authoritative mode = **on**) 
-* **Provisioning brake** (authoritative mode = **on**) - Relations to a receivers are mandatory. It means a identities or roles using as receivers must exists in target IdM. +  * **Provisioning brake**  (authoritative mode = **on**) - Relations to a receivers are mandatory. It means a identities or roles using as receivers must exists in target IdM. 
-* **System scheme** (authoritative mode = **on**) +  * **System scheme**  (authoritative mode = **on**) 
-* **System scheme attributes** (authoritative mode = **on**) +  * **System scheme attributes**  (authoritative mode = **on**) 
-* **Mapping** (authoritative mode = **on**) +  * **Mapping**  (authoritative mode = **on**) 
-* **Attributes mapping** (authoritative mode = **on**) - If a particular attribute maps an EAV attribute to an entity (such as identity), then **the attribute definition is also added to the export**. +  * **Attributes mapping**  (authoritative mode = **on**) - If a particular attribute maps an EAV attribute to an entity (such as identity), then **the attribute definition is also added to the export**. 
-* **Role assigns account in systems** (authoritative mode = **on**, optional = **on**) - Relations between system and roles are exports as optional. It means if some of a role isn't found on target IdM, then is that relation skipped (import will be continue). Within exporting of this relations are role-defined attributes also exported (authoritative mode = **on**, optional = **on**). +  * **Role assigns account in systems**  (authoritative mode = **on**, optional = **on**) - Relations between system and roles are exports as optional. It means if some of a role isn't found on target IdM, then is that relation skipped (import will be continue). Within exporting of this relations are role-defined attributes also exported (authoritative mode = **on**, optional = **on**). 
-* **Sync configuration** (authoritative mode = **on**)+  * **Sync configuration**  (authoritative mode = **on**, optional=**on**) - Sync contract/slice configuration contains relation on a tree-type and tree-node object. If this objects will be not found (by ID or code), then whole configuration of the sync will be skipped (not updated, not created, not deleted). 
 +<note important>Since **10.6.0**: **Sync contract/slice configuration**  contains relation on a organization structure and organization node. If this objects will be not found (by ID or code), then whole **configuration of the sync will be skipped**  (not updated, not created, not deleted)</note> 
 + 
 +<note tip>Virtual systems may be imported with some limitations, see the section **Limitations**.</note>
  
 ==== Application configuration ==== ==== Application configuration ====
Line 95: Line 100:
 **For execute an export action you will need to have:** **For execute an export action you will need to have:**
  
-  * Permission to autocomplete and read, update and create export batch: **Export/Import (IdmExportImport)** | View in select box (autocomplete), Read, Create, Update | BasePermissionEvaluator. +  * Permission to autocomplete and read, update and create export batch: **Export/Import (IdmExportImport)**  | View in select box (autocomplete), Read, Create, Update | BasePermissionEvaluator. 
-  * Permission to see a progress bar: **Scheduler (IdmLongRunningTask)** | Read | BasePermissionEvaluator. +  * Permission to see a progress bar: **Scheduler (IdmLongRunningTask)**  | Autocomplete| BasePermissionEvaluator. 
-  * Permission for read exported object. For example: To export a application configurations you need: **Configuration (app) (IdmConfiguration)** | Read | BasePermissionEvaluator.+  * Permission for read exported object. For example: To export a application configurations you need: **Configuration (app) (IdmConfiguration)**  | Read | BasePermissionEvaluator.
  
 <note important>To successfully export, you must have the **permissions to read all objects that will be exported**. For example, if a role contains business roles and you do not have the permissions to them, then the **export will fail**!</note> <note important>To successfully export, you must have the **permissions to read all objects that will be exported**. For example, if a role contains business roles and you do not have the permissions to them, then the **export will fail**!</note>
Line 103: Line 108:
 **For execute an import action in dry-run mode you will need to have:** **For execute an import action in dry-run mode you will need to have:**
  
-  * Permission to autocomplete and read, update and create export batch: **Export/Import (IdmExportImport)** | View in select box (autocomplete), Read, Create, Update | BasePermissionEvaluator. +  * Permission to autocomplete and read, update and create export batch: **Export/Import (IdmExportImport)**  | View in select box (autocomplete), Read, Create, Update | BasePermissionEvaluator. 
-  * Permission to see a progress bar: **Scheduler (IdmLongRunningTask)** | Read | BasePermissionEvaluator. +  * Permission to see a progress bar: **Scheduler (IdmLongRunningTask)**  | Read | BasePermissionEvaluator. 
-  * Permission to read import log: **Export/Import (IdmImportLog)** | Read | BasePermissionEvaluator.+  * Permission to read import log: **Export/Import (IdmImportLog)**  | Read | BasePermissionEvaluator.
  
 **For execute an import action you will need to have:** **For execute an import action you will need to have:**
  
-  * App admin permission: **App configuration** | Administration (all) | BasePermissionEvaluator. +  * Permission to admin export batch: **Export/Import (IdmExportImport)**  | Administration (all) | BasePermissionEvaluator.
- +
-<note important>Only **super administrator** can run import batches!</note> +
  
 +<note important>Only **import / export administrator**  can run import batches!</note>
  
 ===== Performance tests ===== ===== Performance tests =====
  
-Performance testing took place with using numbers of Systems (specifically CSVDir system) with assigned several roles, set mapping etc. to increase overall number of entities to export. +Performance testing took place with using numbers of Systems (specifically CSVDir system) with assigned several roles, set mapping etc. to increase overall number of entities to export. The goal was to find both particular time of duration and estimation of the export/import complexity.
-The goal was to find both particular time of duration and estimation of the export/import complexity.+
  
   * Testing was performed on NB (i7, 16GB RAM, SSD).   * Testing was performed on NB (i7, 16GB RAM, SSD).
Line 126: Line 128:
 Following table shows number of testing systems and measured times of their export and import. The growth of the operation time is roughly linearly proportional to number of testing systems and by extension overall processed entities. Following table shows number of testing systems and measured times of their export and import. The growth of the operation time is roughly linearly proportional to number of testing systems and by extension overall processed entities.
  
- +^System num^Overall entity num  ^Export time [min]^Import time - Dry run [min]^Import time - Apply changes [min]^Zip size [MB]| 
-^System num ^ Overall entity num  ^ Export time [min] ^ Import time - Dry run [min] ^ Import time - Apply changes [min]^ Zip size [MB] ^ +|111|2691|1|1|3|2,18| 
-| 111 | 2691 | 1 | 1 | 3 | 2,18 | +|201|4851|1|3|10|3,93| 
-| 201 | 4851 | 1 | 3 | 10 | 3,93 |  +|415|9982|2|10|35|8,08| 
-| 415 | 9982 | 2 | 10 | 35 | 8,08 | +|815|19582|4|38|130|15,8|
-| 815 | 19582 | 4 | 38 | 130 | 15,8 |+
  
 ===== Limitations ===== ===== Limitations =====
  
 ==== Cannot be used to update systems / roles that have not been created by the import function ==== ==== Cannot be used to update systems / roles that have not been created by the import function ====
-<note important>Export/Import in version **10.2.0** is primarily intended for export and import of new data (roles / systems). Ie. it is not intended to update the date in the target IdM that was **not created by the import**. The reason for this limitation is the basic property of how import imports existing objects in the target IdM and in the batch. This pairing is based primarily on the identifier, where the same object receives the identifier from the source IdM. This makes it possible to effectively identify an object that is to be updated in the target IdM and to be created as a new one. 
  
-In addition to the basic match of identifiers, a unique code-matching mechanism is implemented in version 10.2.0 (the object must conform to the **Codeable** interface). This mechanism works very well for objects such as **roles**, **identities**,  +<note important>Export/Import in version **10.2.0**  is primarily intended for export and import of new data (roles / systems). Ie. it is not intended to update the date in the target IdM that was **not created by the import**. The reason for this limitation is the basic property of how import imports existing objects in the target IdM and in the batch. This pairing is based primarily on the identifier, where the same object receives the identifier from the source IdM. This makes it possible to effectively identify an object that is to be updated in the target IdM and to be created as a new one.
- **systems**, **catalogs**, those that have unique code.+
  
-However, there are a number of objects that require unique attributes based on two or more attributes. For example, a **scheme of a system** that requires its **name to be unique within the system**. +In addition to the basic match of identifiers, a unique code-matching mechanism is implemented in version 10.2.0 (the object must conform to the **Codeable**  interface). This mechanism works very well for objects such as **roles**, **identities**, **systems**, **catalogs**, those that have unique code. 
-Pairing of these objects is not supported in version **10.2.0**. Trying to update such an object will result in the import attempting to create a **system schema with a duplicate name**, which will result in the entire **import failing**. + 
 +However, there are a number of objects that require unique attributes based on two or more attributes. For example, a **scheme of a system**  that requires its **name to be unique within the system**. Pairing of these objects is not supported in version **10.2.0**. Trying to update such an object will result in the import attempting to create a **system schema with a duplicate name**, which will result in the entire **import failing**.
  
 </note> </note>
  
-==== No skip implemented  ====+==== No skip implemented ==== 
 <note important>Omitting for recalculation of business roles, incompatible roles or provisioning is not implemented in version 10.2.0!</note> <note important>Omitting for recalculation of business roles, incompatible roles or provisioning is not implemented in version 10.2.0!</note>
  
 +==== Virtual systems ====
 +
 +<note tip > Full support of the virtual system export and import has been available since version 11.1. Following workaround should not be necessary anymore. </note>
 +
 +When importing a virtual system, you must make some further adjustments so the imported system behaves in the same way as the exported system. Especially if you use the "rights" attribute or if you changed or [[:tutorial:adm:virtual_system_-_adding_or_deleting_managed_extended_attributes#deleting_attributes_on_virtual_systems|removed some of the default attributes]]. After you import the system:
  
-==== Future improvements ==== +  * Go to **Settings → Form definitions**, search for the definition of type *VsAccount* and name containing the name of your system. 
-<note tip >**Modularity** - Allow to export a single batch across multiple modules.</note>+  * Go to **Form attributes**, remove the attributes that are superfluous. For the attribute "rights", switch the checkbox "multivalued".
  
 +More info [[https://redmine.czechidm.com/issues/2550|here]].
  
  
  • by svandav