Connector - Implementation of a new ConnID connector

In this tutorial, we will see how to implement connector. Our example is CSV Connector.

Other option is to use this command to generate skeleton for new connector. You don't need to clone the repository and this should work, because the connector-archetype artifact is available in standard maven repository.

mvn archetype:generate '-DarchetypeGroupId=net.tirasa.connid' '-DarchetypeArtifactId=connector-archetype' '-DarchetypeRepository=http://repo1.maven.org/maven2' '-DarchetypeVersion=1.4.3.0'

After this you will be prompted to fill additional information about your connector. See picture in Second option how to create connector

We do not have start whole project from scratch. What we need to is:

• Download or clone archetype - https://github.com/Tirasa/ConnId
• Move to the location of downloaded archetype (where pom.xml is situated)
• Now in this directory just type - 'mvn install'

• After we have this setup, we can start to generate the schema. Just type 'mvn archetype:generate' in terminal where you want your connector
• We have to go through point-to-point creation of our connector as you can see in the picture under Few other steps to go through in the picture below in Second option how to create connector.

• Here we have to find number of the 'net.tirasa'

• Few other steps to go through in the picture below

The basic skeleton for our connector is now done. We can open it in our IDEA.

When you have skeleton for the new connector from artifact, you need to created two files if you have plans to edit any file which was generated. Your new connector will be under CDDL and Apache licence. Requirements are changelog file and licence file.

In root directory for you connector create:

• LICENCE - example https://github.com/bcvsolutions/moodle-connector/blob/develop/LICENSE
• CHANGELOG.md - example https://github.com/bcvsolutions/moodle-connector/blob/develop/CHANGELOG.md

Now everything is generated so we can start to implement the connector. First, we need to change the name of our Sample classes. Let's say we change these names to CSVConnConnector, CSVConnConfiguration and CSVConnFilter.

It is important to know, how to work with connectors in CzechIdM. We need to follow steps in Idm to implement correctly our connector. First what we do, when we want the new connector to be connected to Idm, is to import this connector into CzechIdm. For our usage, we can just build our connector with maven goal: mvn package where the connector is located. Then build also Idm and before running just paste connector's bundle jar file into tomcat/webapps/idm-backend/WEB-INF/lib/.

Also we need to use predefined path: "eu.bcvsolutions.idm.connector" as a path to main connector classes. This path is added in module-ic.properties file.

# Defines packages where will module search for available connectors
# You can use multivalues (ic.localconnector.packages=net.tirasa.connid.bundles.db,net.tirasa.connid.bundles.ldap)
# You can define only start of package
ic.localconnector.packages=net.tirasa.connid,eu.bcvsolutions.idm.vs.connector,**eu.bcvsolutions.idm.connector**

First of all, we will look at CSVConnConfiguration.

public class CSVConnConfiguration extends AbstractConfiguration {
 
    /**
     * Separator of CSV file for each column
     */
    private String separator = ";";
 
    /**
     * encoding of CSV file
     */
    private String encoding = "UTF-8";
 
    /**
     * path to CSV file
     */
    private String sourcePath;
 
    /**
     * boolean if CSV file includes header or not
     */
    private boolean includesHeader = false;
 
    /**
     * if file doesn't include header, it has to be set before parse
     */
    private String[] header;
 
    /**
     * Have to be set identifier = __UID__
     */
    private String uid;
 
    /**
     * Have to be set __NAME__
     */
    private String name;
 
    @ConfigurationProperty(displayMessageKey = "separator.display",
            helpMessageKey = "separator.help", order = 1)
    public String getSeparator() {
        return separator;
    }
 
    public void setSeparator(String separator) {
        this.separator = separator;
    }
 
    @ConfigurationProperty(displayMessageKey = "encoding.display",
            helpMessageKey = "encoding.help", order = 2)
    public String getEncoding() {
        return encoding;
    }
 
    public void setEncoding(String encoding) {
        this.encoding = encoding;
    }
 
    @ConfigurationProperty(displayMessageKey = "includesHeader.display",
            helpMessageKey = "includesHeader.help", order = 3)
    public boolean isIncludesHeader() {
        return includesHeader;
    }
 
    public void setIncludesHeader(boolean includesHeader) {
        this.includesHeader = includesHeader;
    }
 
    @ConfigurationProperty(displayMessageKey = "sourcePath.display",
            helpMessageKey = "sourcePath.help", order = 4)
    public String getSourcePath() {
        return sourcePath;
    }
 
    public void setSourcePath(String sourcePath) {
        this.sourcePath = sourcePath;
    }
 
    @ConfigurationProperty(displayMessageKey = "header.display",
            helpMessageKey = "header.help", order = 5)
    public String[] getHeader() {
        return header;
    }
 
    public void setHeader(String[] header) {
        this.header = header;
    }
 
    @ConfigurationProperty(displayMessageKey = "uid.display",
            helpMessageKey = "uid.help", order = 6)
    public String getUid() {
        return uid;
    }
 
    public void setUid(String uid) {
        this.uid = uid;
    }
 
    @ConfigurationProperty(displayMessageKey = "name.display",
            helpMessageKey = "name.help", order = 7)
    public String getName() {
        return name;
    }
 
    public void setName(String name) {
        this.name = name;
    }
 
    @Override
    public void validate() {
 
        if (StringUtil.isBlank(separator) || separator == null) {
            throw new ConfigurationException("separator must not be blank!");
        }
 
        if (StringUtil.isBlank(encoding) || encoding==null) {
            throw new ConfigurationException("encoding must not be blank!");
        }
 
        if (StringUtil.isBlank(sourcePath) || sourcePath==null) {
            throw new ConfigurationException("sourcePath must not be blank!");
        }
 
        if(!includesHeader && header==null){
            throw new ClassCastException("File doesn't include header, but header is not set!");
        }
 
        if(StringUtil.isBlank(uid) || uid==null){
            throw new ConfigurationException("uid must not be blank!");
        }
    }
 
}

As you can see in the code it is just pile of getters and setters.

• Variables are basically properties which are filled in the frontend. There need to be setters and getters for each.
• Annotations are mostly necessary for name tag and help tag. Both can be then filled in resource/Message.properties.
• If you need to pre-fill some of the variables, just fill it here.
• Method validate() is method for Test Connector in connector's configuration. Also all properties you can see there.
• Order in the annotation is just for order in frontend page.

This class is the brain of the connector.

Initialize configuration filled by user.

@Override
public void init(final Configuration configuration) {
    this.configuration = (CSVConnConfiguration) configuration;
    LOG.ok("Connector {0} successfully inited", getClass().getName());
}

Now we need to somehow generate the schema.

Schema function generates the new schema in IDM. The main problem of this function is to analyze all columns and find identifier. In our example, we have to see the header of the CSV file. It can be found externally or we have to see the header in the file.

@Override
public Schema schema() {
    return new CreateSchema(configuration).generateSchema(this);
}

/**
 * This class creates schema from given CSV file (Configuration path).
 * @author Marek Klement
 */
public class CreateSchema {
 
	private final CSVConnConfiguration conf;
	private static final Log LOG = Log.getLog(CreateSchema.class);
 
	public CreateSchema(CSVConnConfiguration conf){
		this.conf = conf;
	}
 
	public Schema generateSchema(Connector connector){
		String[] header = null;
		final CSVReader reader;
		final CSVParser parser;
		try {
			parser = new CSVParserBuilder()
					.withSeparator(conf.getSeparator().charAt(0))
					.withIgnoreQuotations(true)
					.build();
			reader = new CSVReaderBuilder(new FileReader(conf.getSourcePath()))
					.withCSVParser(parser)
					.build();
			Iterator<String[]> it = reader.iterator();
			if(conf.isIncludesHeader()){
				header = findHeader(it);
			} else if(!conf.isIncludesHeader() && conf.getHeader()!=null){
				checkLength(it,conf.getHeader().length, 1);
				header = conf.getHeader();
			} else {
				throw new IllegalArgumentException("CREATESCHEMA - Wrong header found!");
			}
			reader.close();
		} catch (FileNotFoundException e) {
			e.printStackTrace();
		} catch (IOException e) {
			e.printStackTrace();
		}
		return createSchema(connector,header);
	}
 
	private String[] findHeader(Iterator<String[]> it){
		String[] header;
		if(it.hasNext()){
			header = it.next();
			checkLength(it,header.length,2);
		} else {
			throw new IllegalArgumentException("CREATESCHEMA - CSV file is blank, cannot resolve header!");
		}
		LOG.ok("CREATESCHEMA - Header found successfully!");
		return header;
	}
 
	private void checkLength(Iterator<String[]> it, int length, int i){
		while (it.hasNext()){
			String[] next = it.next();
			if(next.length!=length){
				throw new IllegalArgumentException("CREATESCHEMA - At "+i+". line is "+next.length+" items, but expect "+length);
			}
			++i;
		}
		LOG.ok("CREATESCHEMA - All lines have same amount of items!");
	}
 
	private Schema createSchema(Connector connector, String[] header){
		SchemaBuilder sb = new SchemaBuilder(connector.getClass());
		final Set<AttributeInfo> attributeInfos = new HashSet<>();
		if(conf.getName()==null){
			conf.setName(conf.getUid());
		}
		for(String column : header) {
			final AttributeInfoBuilder attributeInfoBuilder = new AttributeInfoBuilder();
			if(column.equals(conf.getUid()) || column.equals(conf.getName())){
				attributeInfoBuilder.setRequired(true);
 
			}
			attributeInfoBuilder.setName(column);
			attributeInfoBuilder.setCreateable(true);
			attributeInfoBuilder.setUpdateable(true);
			attributeInfos.add(attributeInfoBuilder.build());
		}
		sb.defineObjectClass(ObjectClass.ACCOUNT_NAME, attributeInfos);
		LOG.ok("CREATESCHEMA - Schema was created successfully!");
		return sb.build();
	}
}

This method is the call for creating new item in the CSV file. It should find the end of the CSV file and put the new item with all attributes on the bottom of the file.

    @Override
    public Uid create(
            final ObjectClass objectClass,
            final Set<Attribute> createAttributes,
            final OperationOptions options) {
        try {
            return new CreateItem(configuration).createItem(objectClass,createAttributes,options);
        } catch (IOException e) {
            e.printStackTrace();
        }
        throw new IllegalArgumentException("Found error in create item!");
    }

/**
 * Class for creating items in CSV file from system
 * @author Marek Klement
 */
public class CreateItem extends Operations {
 
	private static final Log LOG = Log.getLog(CreateItem.class);
 
	public CreateItem(CSVConnConfiguration conf) {
		super(conf);
	}
 
	public Uid createItem(ObjectClass objectClass, Set<Attribute> createAttributes, OperationOptions options) throws IOException {
		String name = findName(createAttributes);
		if(name == null || StringUtil.isBlank(name)){
			throw new IllegalArgumentException("CREATEITEM - Name was not provided!");
		}
		if(userExists(name)){
			throw new IllegalArgumentException("CREATEITEM - User already exists!");
		}
		addItem(createAttributes);
		LOG.ok("CREATEITEM - Item with identificator "+name+" was created successfully!");
		return new Uid(name);
	}
 
	private void addItem(Set<Attribute> createAttributes) throws IOException {
		final CSVWriter writer = new CSVWriter(new FileWriter(conf.getSourcePath(),true), conf.getSeparator().charAt(0),CSVWriter.NO_QUOTE_CHARACTER,CSVWriter.NO_ESCAPE_CHARACTER,"\n");
		String[] nextLine = createNewLine(createAttributes);
		writer.writeNext(nextLine);
		writer.close();
	}
}

Operation delete is almost same as create(). Just with the difference that instead of creating new item we simply delete the item with given Uid from CSV file.

    @Override
    public void delete(
            final ObjectClass objectClass,
            final Uid uid,
            final OperationOptions options) {
        try {
            new DeleteItem(configuration).deleteItem(objectClass ,uid, options);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

/**
 * Class for delleting items in CSV file
 * @author marek
 */
public class DeleteItem extends Operations {
 
	private static final Log LOG = Log.getLog(CreateItem.class);
 
	public DeleteItem(CSVConnConfiguration conf) {
		super(conf);
	}
 
	public Uid deleteItem(ObjectClass objectClass, Uid uid, OperationOptions options) throws IOException {
		if(uid == null){
			throw new IllegalArgumentException("DELETEITEM - Identifier must not be null!");
		}
		String name = uid.getUidValue();
		if(name == null || StringUtil.isBlank(name)){
			throw new IllegalArgumentException("DELETEITEM - Name was not provided!");
		}
		if(!userExists(name)){
			throw new IllegalArgumentException("DELETEITEM - Nothing to delete!");
		}
		removeItem(name);
		LOG.ok("DELETEITEM - Item with identificator "+name+" was deleted successfully!");
		return new Uid(name);
	}
 
	private void removeItem(String name) throws IOException {
		CSVParser parser = new CSVParserBuilder()
				.withSeparator(conf.getSeparator().charAt(0))
				.withIgnoreQuotations(true)
				.build();
		CSVReader reader = new CSVReaderBuilder(new FileReader(conf.getSourcePath()))
				.withCSVParser(parser)
				.build();
		Iterator<String[]> it = reader.iterator();
		List<String[]> buffer = new ArrayList<>();
		int nameNumber = getNameNumber(headerFound);
		while (it.hasNext()){
			String[] line = it.next();
			if(!line[nameNumber].equals(name)){
				buffer.add(line);
				System.out.println(line);
			}
		}
		reader.close();
		CSVWriter writer = new CSVWriter(new FileWriter(conf.getSourcePath()), conf.getSeparator().charAt(0),CSVWriter.NO_QUOTE_CHARACTER,CSVWriter.NO_ESCAPE_CHARACTER,"\n");
		writer.writeAll(buffer);
		writer.close();
	}
}

This method takes care of updating items in CSV file. On the connected system, it just finds right identifier value and then updates all attributes.

    @Override
    public Uid update(
            final ObjectClass objectClass,
            final Uid uid,
            final Set<Attribute> replaceAttributes,
            final OperationOptions options) {
 
        try {
            return new UpdateItem(configuration).updateItem(objectClass,uid,replaceAttributes,options);
        } catch (IOException e) {
            e.printStackTrace();
        }
        throw new IllegalArgumentException("Found error in update item!");
    }

/**
 * Class for update record in CSV file
 * @author Marek Klement
 */
public class UpdateItem extends Operations{
 
	private static final Log LOG = Log.getLog(CreateItem.class);
 
	public UpdateItem(CSVConnConfiguration conf) {
		super(conf);
	}
 
	public Uid updateItem(ObjectClass objectClass, Uid uid, Set<Attribute> updateAttributes, OperationOptions options) throws IOException {
		if(uid == null){
			throw new IllegalArgumentException("UPDATEITEM - Identifier must not be null!");
		}
		String name = uid.getUidValue();
		if(name == null || StringUtil.isBlank(name)){
			throw new IllegalArgumentException("UPDATEITEM - Name was not provided!");
		}
		if(!userExists(name)){
			throw new IllegalArgumentException("UPDATEITEM - User doesn't exists!");
		}
		if(updateAttributes==null){
			throw new IllegalArgumentException("UPDATEITEM - UpdateAttributes were not provided!");
		}
		update(updateAttributes, name);
		LOG.ok("UPDATEITEM - Item with identificator "+name+" was updated successfully!");
		return new Uid(name);
	}
 
	private void update( Set<Attribute> updateAttributes, String name) throws IOException {
		int nameNumber = getNameNumber(headerFound);
		CSVParser parser = new CSVParserBuilder()
				.withSeparator(conf.getSeparator().charAt(0))
				.withIgnoreQuotations(true)
				.build();
		CSVReader reader = new CSVReaderBuilder(new FileReader(conf.getSourcePath()))
				.withCSVParser(parser)
				.build();
		Iterator<String[]> it = reader.iterator();
		List<String[]> buffer = new ArrayList<>();
		while (it.hasNext()){
			String[] line = it.next();
			if(!line[nameNumber].equals(name)){
				buffer.add(line);
			} else {
				line = createNewLine(updateAttributes);
				buffer.add(line);
			}
		}
		reader.close();
		CSVWriter writer = new CSVWriter(new FileWriter(conf.getSourcePath()), conf.getSeparator().charAt(0),CSVWriter.NO_QUOTE_CHARACTER,CSVWriter.NO_ESCAPE_CHARACTER,"\n");
		writer.writeAll(buffer);
		writer.close();
	}
}

This function is mostly brain of the connector. Basically, it reads all items in our CSV file and then it creates objects for the system. It all depends on filter we use. This function can be called:

• as Sync, where we use to filter
• with no filter where we just create new identities

    @Override
    public void sync(
            final ObjectClass objectClass,
            final SyncToken token,
            final SyncResultsHandler handler,
            final OperationOptions options) {
        LOG.info("Starting SYNC");
        CSVConnFilter filter = createSyncFilter(token, configuration.getSyncTokenColumn());
        ResultsHandler resultsHandler = connectorObject -> {
 
            SyncToken newToken = new SyncToken(connectorObject.getAttributeByName(configuration.getSyncTokenColumn()).getValue().get(0));
            SyncDeltaBuilder builder = new SyncDeltaBuilder();
			builder.setObject(connectorObject)
                    .setToken(newToken)
                    //TODO operation DELETE
                    .setDeltaType(SyncDeltaType.CREATE_OR_UPDATE)
					.setObjectClass(objectClass);
			return handler.handle(builder.build());
		};
        executeQuery(objectClass, filter, resultsHandler, options);
    }
 
    private CSVConnFilter createSyncFilter(SyncToken token, String syncTokenColumn) {
        final CSVConnFilter filter = new CSVConnFilter(CSVConnFilter.Operation.GT, syncTokenColumn, token.getValue());
        return filter;
    }

    @Override
    public void executeQuery(
            final ObjectClass objectClass,
            final CSVConnFilter query,
            final ResultsHandler handler,
            final OperationOptions options) {
        LOG.info("Starting executeQuery");
        List<ConnectorObject> result = new CreateObjectsFromCSV(this.configuration, query).parse();
        result.forEach(handler::handle);
    }

And now main functionality of this method:

/**
 * Class for holding functions to parse CSV into object
 * @author Marek Klement
 */
public class CreateObjectsFromCSV {
 
	private final CSVConnConfiguration conf;
	private final CSVConnFilter filter;
	private static final Log LOG = Log.getLog(CreateObjectsFromCSV.class);
 
	public CreateObjectsFromCSV(CSVConnConfiguration conf, CSVConnFilter filter){
		this.conf = conf;
		this.filter = filter;
	}
 
	public List<ConnectorObject> parse(){
		List<ConnectorObject> items = null;
		final CSVReader reader;
		final CSVParser parser;
		try {
			parser = new CSVParserBuilder()
					.withSeparator(conf.getSeparator().charAt(0))
					.withIgnoreQuotations(true)
					.build();
			reader = new CSVReaderBuilder(new FileReader(conf.getSourcePath()))
					.withCSVParser(parser)
					.build();
			String[] header;
			if(conf.isIncludesHeader()){
				header = reader.readNext();
				conf.setHeader(header);
			} else {
				header = conf.getHeader();
			}
			items = new LinkedList<>();
			Iterator<String[]> it = reader.iterator();
			if(conf.getName()==null){
				conf.setName(conf.getUid());
			}
			LOG.info("EXECUTEQUERRY - started to read CSV");
			while (it.hasNext()){
				String[] item = it.next();
				final ConnectorObject obj = transform(header,item,conf.getName(),conf.getUid());
				if(filter!=null) {
					if (filter.evaluate(obj)) {
						items.add(obj);
						LOG.ok("Object "+obj.getName().getNameValue()+" was added successfully!");
					}
				} else {
					items.add(obj);
				}
			}
			LOG.ok("All objects were created successfully!");
			reader.close();
		} catch (FileNotFoundException e) {
			e.printStackTrace();
		} catch (IOException e) {
			e.printStackTrace();
		} finally {
			//reader.close();
		}
		return items;
	}
 
	private ConnectorObject transform(String [] header, String [] line, String name, String uid) {
		ConnectorObjectBuilder builder = new ConnectorObjectBuilder();
 
		for (int i = 0; i < header.length; i++) {
			String head = header[i];
			String lin = line[i];
			if(head.equals(name)){
				builder.setName(lin);
			}
			if(head.equals(uid)){
				builder.setUid(lin);
			}
			builder.addAttribute(head, lin);
		}
		return builder.build();
	}
 
}

My worst problem was with getting know to filter. A filter is a tool for Synchronization mostly. It can spot last changed an item in CSV file and ignore all before. Lets see my Filter and FilterTranslator:

public class CSVFilterTranslator extends AbstractFilterTranslator<CSVConnFilter> {
 
	private static final Log LOG = Log.getLog(CSVFilterTranslator.class);
 
	@Override
	protected CSVConnFilter createEqualsExpression(final EqualsFilter filter, final boolean not) {
		LOG.info("CSVFilterTranslator -- createEqualsExpression");
		return new CSVConnFilter(CSVConnFilter.Operation.EQ, filter.getAttribute().getName(), filter.getName());
	}
 
 
	@Override
	protected CSVConnFilter createGreaterThanExpression(GreaterThanFilter filter, boolean not) {
		LOG.info("CSVFilterTranslator -- createGreaterThanExpression");
		return new CSVConnFilter(CSVConnFilter.Operation.GT, filter.getAttribute().getName(), filter.getValue());
	}
}

and filter itself:

public class CSVConnFilter {
 
	private static final Log LOG = Log.getLog(CSVConnFilter.class);
 
	public enum Operation {
		EQ, GT
	}
 
	private final Operation operation;
	private final String attributeName;
	private final Object attributeValue;
 
	public CSVConnFilter(Operation operation, String attributeName, Object attributeValue){
		this.operation = operation;
		this.attributeName = attributeName;
		this.attributeValue = attributeValue;
	}
 
 
	public boolean evaluate(ConnectorObject obj) {
		final Attribute attribute = obj.getAttributeByName(attributeName);
		boolean ret;
		LOG.info("ATTRIBUTE "+attribute.getName()+" WITH value "+attribute.getValue()+ " compare to ATTRIBUTE "+attributeValue);
		if (attribute == null) {
			return false;
		}
		switch (operation) {
			case EQ:
				//TODO: multivalued
				ret = attributeValue.equals(attribute.getValue().get(0));
				return ret;
			case GT:
				int bh = attributeValue.toString().compareTo(attribute.getValue().get(0).toString());
				ret = bh<0;
				return ret;
			default:
				return true;
		}
 
	}
 
	public Operation getOperation() {
		return operation;
	}
 
	public String getAttributeName() {
		return attributeName;
	}
 
	public Object getAttributeValue() {
		return attributeValue;
	}
 
}

After everything is written we should add our new connector into IDM. It might look like its hard but actually, it is easy. We just add dependency into app module in CzechIdM app. So for our example, this is what you should add when we need to use CSV connector:

<dependency>
   <groupId>eu.bcvsolutions</groupId>
   <artifactId>csv-connector</artifactId>
   <version>1.3.6-SNAPSHOT</version>
</dependency>

If you want to log for example INFO messages from connectors you have to add following property to your active profile in logback-spring.xml:

<logger name="eu.bcvsolutions.idm.connector" level="INFO"/>

name means the package where you want to setup logs and level sets up types of logs. For setting of WARN messages you can copy this property and just change level to "WARN". Info messages are switched on in default profile which is used in production, but usually switched off for development.