====== Auditing ======
{{tag> audit transaction}}

<note important>TODO: Introduction</note>

Auditing of entities is done by:
  * [[http://hibernate.org/orm/envers/|Hibernate envers]] - Historizes data. Audit logs are saved in tables with ''_a'' suffix according to a specific entity. Auditing is linked to all the basic entities, except for entities which cannot be audited. ([[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/entity/IdmAudit.java|IdmAudit]], [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/entity/IdmPassword.java|IdmPassword]], etc.).
  * [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/repository/listener/IdmAuditListener.java|IdmAuditListener]] - The individual tables are linked by the main auditing entity = [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/entity/IdmAudit.java|IdmAudit]], over which an agenda is built which can be searched via all audited entities. [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/repository/listener/IdmAuditListener.java|IdmAuditListener]] handles filling in of this agenda. It listens to changes over all the entities being audited and creates references in tables with historical data created via [[http://hibernate.org/orm/envers/|hibernate envers]].
  * [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-api/src/main/java/eu/bcvsolutions/idm/core/api/repository/listener/AuditableEntityListener.java|AuditableEntityListener]] - handles filling in of the audit attributes in entities (date of creation, modification, etc.).
  * [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/repository/listener/AuditableInterceptor.java|AuditableInterceptor]] - ensures that the once filled-in date of creation is not overwritten when the log is modified.

<note important>IdmAudit is the only entity in the CzechIdMng system which has Long as its primary key. All the other entities have UUID. This fact is unchangeable and needed by Envers.</note>

====== New entity, how to enable auditing ======
There is a simple way of "enabling" versing for an entity in [[http://hibernate.org/orm/envers/|Hibernate envers]]. It is done simply by writing the annotation [[https://docs.jboss.org/hibernate/orm/3.5/api/org/hibernate/envers/Audited.html|@Audited]] before the entity's definition. We **DO NOT USE** this way in the CzechIdMng system. Instead, we write this annotation to the individual entity attributes which we want to audit. For example: 
<code java>
 @Audited
 @NotNull
 @Column(name = "disabled", nullable = false)
 private boolean disabled = false;
</code>


===== IdmAuditService =====
[[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/service/api/IdmAuditService.java|IdmAuditService]] serves as a mediator between [[http://hibernate.org/orm/envers/|Hibernate envers]] and [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/entity/IdmAudit.java|IdmAudit]]. The service provides an access to the audit metadata ([[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/entity/IdmAudit.java|IdmAudit]]) and the versions themselves. The basic methods of this service are divided into ''findRevision'' and ''findVersion''. The description of their differences can be found below.

==== Revision ====
Revisions in the CzechIdMng system are basic metadata about the individual revisions. The metadata is propagated to the front-end to the agenda itself.
{{ :navrh:audit01.png |}}
Using revisions, we are able to ascertain the following metadata:
  * entityId //(UUID of the entity involved in the revision)//
  * modification //(type of modification, adding, removing, modification)//
  * type //(type of the audited entity, class name)//
  * changedAttributes //(a chain which is, in case of modification, filled in with the names of the columns which have been changed)//
  * modifier //(username of the identity, which has made a change, adding or removing)//
  * modifierId //(UUID of the identity, which has made a change, adding or removing)//

The ''findRevision'' methods are able to return exactly one revision for the given class, entity ID and revision ID, and also all the revisions for the class and entity ID. Using the ''getPreviousRevision'' method, it is possible to get the previous revision according to the ID of the revision.

==== Version ====

A version in the CzechIdMng system is a [[https://en.wikipedia.org/wiki/Snapshot_(computer_storage)|snapshot]] of the entity at a given time. Methods handling versions do not allow getting all the versions for the entity. Instead, they support getting one version ''getVersion'', the previous version ''getPreviousVersion'', and the difference //(diff)// between the versions ''getDiffBetweenVersion''. If we need to ascertain the last entity version, we can use the ''getLastVersionNumber'' method.

If we do not know the exact entity type //(class)//, an object will be returned as the version. If we want to get the individual values from this unidentifiable type, we can use the ''getValuesFromVersion'' method. If we need a list of the changed attributes between the versions, we can use the ''getDiffBetweenVersion'' method.

A complete list of the audited entities can be acquired by using the ''getAllAuditedEntitiesNames'' method.

==== Return version including version ====

If you want return audit revision including version of entity you must set boolean flag **withVersion**. About this feature you can read this [[devel:documentation:audit:dev:auditwithversion|documentation]].

==== Graphical description of the differences between revision and version ====

{{ :navrh:audit02.png |}}

==== Exact description of the individual methods ====

<note tip>Some method in this service has typing error. For example **getPreviosAuditVersion** sorry for this complication we cannot update these names :( </note>

The current version of these methods can be found in the [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/service/api/IdmAuditService.java|IdmAuditService]] interface. The most important methods have been chosen.

=== <T> T findRevision(Class<T> classType, UUID entityId, Long revisionId) ===
The method returns one revision, if it exists, or it returns an exception [[http://docs.jboss.org/hibernate/orm/4.0/javadocs/org/hibernate/envers/exception/RevisionDoesNotExistException.html|RevisionDoesNotExistException]]
\\
\\
=== <T> List<IdmAudit> findRevisions(Class<T> classType, UUID entityId) ===
The method returns all revisions for the entity. If no revision is found, an empty list is returned.
\\
\\
=== <T> T getPreviousVersion(T entity, Long currentRevId) ===
It returns the exact type of version, to the previous revision. The previous version is established using the revision ID.
\\
\\
=== Object getPreviousVersion(Long currentRevId) ===
It returns the previous entity version for the id revision. The version type cannot be determined easily at this moment.
\\
\\
=== <T> T getPreviousVersion(Class<T> entityClass, UUID entityId, Long currentRevId) ===
The method returns the exact version type which is determined by revision ID and entity ID.
\\
\\
=== <T> Number getLastRevizionNumber(Class<T> entityClass, UUID entityId) ===
The method returns the last available revision number.
\\
\\
=== <T> List<String> getNameChangedColumns(Class<T> entityClass, UUID entityId, Long currentRevId, T currentEntity) ===
It returns a list of changed entity attributes. Note that only attributes with the [[https://docs.jboss.org/hibernate/orm/3.5/api/org/hibernate/envers/Audited.html|@Audited]] annotation are accepted.
\\
\\
=== List<String> getAllAuditedEntitiesNames() ===
A list of all entities containing at least one [[https://docs.jboss.org/hibernate/orm/3.5/api/org/hibernate/envers/Audited.html|@Audited]] annotation is returned.
\\
\\
=== Map<String, Object> getDiffBetweenVersion(String clazz, Long firstRevId, Long secondRevId) ===
A map is returned, where the key is the name of the attribute which has been changed in the second revision. The value is then the changed value from the second version.
\\
\\
=== Map<String, Object> getValuesFromVersion(Object revisionObject) ===
If we do not know the exact version type, we can get it using ''PropertyUtils.getPropertyDescriptor''([[http://commons.apache.org/proper/commons-beanutils/apidocs/org/apache/commons/beanutils/PropertyUtils.html#getPropertyDescriptors-java.lang.Object-|getPropertyDescriptor]])
\\
\\

===== REST endpoints =====
The information about audit is located on the ''/audits'' endpoint. The right for endpoints ''AUDIT\_WRITE'' does not exist for audit. All the other information about audit are secured using ''AUDIT\_READ''. If a user owns a role with this permission, he is able, for example, to display data about users, to which he does not have to have any rights at all. This situation is desired.

Apart from the basic endpoints, the REST controller is extended by these endpoints.

==== /{revId}/diff/previous  ====
The previous revisions is returned to the revision ID. If it does not exist, http status 404 is returned.

==== /{firstRevId}/diff/{secondRevId} ====
The endpoint compares the difference between two versions of revisions.

===== Front-end =====

==== Dynamic audit detail ====

The same detail has been created for all entities to display audit detail. This detail enables displaying and comparison of two revisions including the difference of their version values.

===== IdmAuditListener =====

The listener [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/repository/listener/IdmAuditListener.java|IdmAuditListener]] handles filling in of revision metadata.

At the moment, it performs these operations:
  * Sets the attributes: type (class name), modification, modifier name and ID, audited entity ID.
If it is the MOD (modification) type modification, comparison with the previous revision is done and the attribute ''changedColumns'' is given the names of the changed attributes.

===== User transaction =====

**User** transaction is started, when operation is executed by rest controller (''StartUserTransactionFilter'') or by scheduled task (''LongRunningTaskManager''). Each user transaction (~operation) has uniquie uuid identifier and all entities, which are modified in this user transaction has this transaction udentifier persisted in ''transactionId'' attribute. This atttribute can be used for filtering - e.g. in audit for entites, entity events. If event is executed asynchronously, then original user transaction is used and event processing continues in the same transaction identifier.

<note tip>If we want to know, which entities was modified in one user transaction, we can use LRT (e.g. from running synchronization) and use them in audit for entities agenda.</note>

<note tip>Transcation id can be shown in UI by [[..:..:application_configuration:dev:backend#applicationserver|configuration]].</note> 


===== Temporary errors =====

==== Two instances of saving in one transaction ====

During synchronization (and not only there), a problem has been found with multiple saving of one entity in one transaction. For Envers, one transaction means one revision (the revision can then contain multiple versions/snapshots of various entities. If a creation of an entity is done in a transaction and then, after saving it, it is updated, the revision will contain two versions/snapshots of the entity. The **problem** comes when we want to fill in the list of changed attributes in the version because the previous version has not been committed yet.

Temporary solution: The functionality of changed attributes is temporarily disabled, the list of changed attributes can only be seen in the detail.


==== Future development ====
  * adding information about the current PPV in the detail of identity audit
  * adding the stress test


====== Audit for identities ======
To make operating with the audit for identities better (for example filtering by username), a FE agenda was created for searching identities with their relationship. 6 new columns were added for the entity [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/entity/IdmAudit.java#L112-L128|IdmAudit]] that are filled in the [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/repository/listener/IdmAuditListener.java#L96-L104|IdmAuditListener]]. These columns are filled only when the audited entity implements the interface [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-api/src/main/java/eu/bcvsolutions/idm/core/api/domain/AuditSearchable.java|AuditSearchable]] (for example [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/entity/IdmIdentityRole.java|IdmIdentityRole]]).

=== Columns description ===
  * **ownerId** - unique identifier of the owner of this entity,
  * **ownerCode** - human readable identifier of the owner of this entity (example username),
  * **ownerType** - class of the owner,
  * **subOwnerId** - unique identifier of the owner of this entity,
  * **subOwnerCode** - human readable identifier of the owner of this entity,
  * **subOwnerType** - class of the sub owner.

The use of Sub owner columns is recommended only for intersection tables. The example of filling the attributes ([[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/model/entity/IdmIdentityRole.java|IdmIdentityRole]]):
<code java>
@Override
public String getOwnerId() {
	return this.getIdentityContract().getIdentity().getId().toString();
}

@Override
public String getOwnerCode() {
	return this.getIdentityContract().getIdentity().getCode();
}

@Override
public String getOwnerType() {
	return IdmIdentity.class.getName();
}

@Override
public String getSubOwnerId() {
	return this.getRole().getId().toString();
}

@Override
public String getSubOwnerCode() {
	return this.getRole().getCode();
}

@Override
public String getSubOwnerType() {
	return IdmRole.class.getName();
}
</code>

===== Endpoint for getting entities with their relations =====
Endpoint for getting some entity with their relation is **/audit/search/entity** ([[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/rest/impl/IdmAuditController.java#L72-L91|IdmAuditController]]). The parameter **entity** is required, the value is the name of the class (service method [[https://github.com/bcvsolutions/CzechIdMng/blob/develop/Realization/backend/core/core-impl/src/main/java/eu/bcvsolutions/idm/core/audit/service/impl/DefaultAuditService.java#L776-L788|DefaultAuditService (findEntityWithRelation)]])

===== Find previous version =====

<note important>Check result of this method while you use it on project.</note>

Method ''auditService.findPreviousRevision'' can return different entity type than original revision (At least in version 9.7.11). This is made by saving more entities in one transaction. For example in synchronization IdmContractSlice. But exists better solution than use this method -> create Envers audit criteria.