Guidelines

This article contains the main building blocks for usage of JDUT.

For quick example, see Examples.


1. Java API

This section describes the usage of pure-Java API for building/cleaning data

1.1. Set-up data grain

DataGrain

Used to set-up values and definitions of data

Supplier<?>

The value of field could be a supplier, which working as lazy-evaluation

See DataGrainTest for full example

Quick example:

import idv.mikelue.jdut.datagrain.DataGrain;

DataGrain dataGrainByDefine = DataGrain.build(
	/**
	 * Defines the table meta-data
	 * The columns would be indexed by sequence of builds
	 */
	tableSchemaBuilder -> {
		tableSchemaBuilder
			.name("tab_1") // Mandatory
			.keys("col_1", "col_2"); // Used by DELETE operation
	},
	// :~)
	/**
	 * Adds the data of table
	 */
	rowsBuilder -> {
		rowsBuilder.implicitColumns(
			"col_1", "col_2", "col_3", "col_4"
		)
		/**
		 * Implicit data
		 */
		.addValues(10, "CC-1", () -> "AC-2", null)
		.addValues(11, "CC-1", () -> "AC-2", null)
		// :~)
		/**
		 * Explicit data
		 */
		.addFields(
			rowsBuilder.newField("col_1", 12),
			rowsBuilder.newField("col_2", 13),
			rowsBuilder.newField("col_3", () -> "AN-78")
			rowsBuilder.newField("col_4", () -> "AN-78")
		);
		// :~)
	}
	// :~)
);

1.2. Execute operator with data grain

DataConductor

The conductor depending on DataSource

DataGrainOperator

The code(SQL) executes on database with DataGrain

DefaultOperatorFactory

The factory for fetching operator(may be vendor-specific) by name.

import guru.mikelue.jdut.DataConductor;
import guru.mikelue.jdut.operation.DefaultOperators;

/**
 * dataSource - The initialized data source
 *
 * This conductor would load schema of database to complement the complete information of column.
 */
DataConductor dataConductor = DataConductor.build(dataSource);

/**
 * Executes the "INSERT"(defined in OperatorFactory) with data grain
 */
dataConductor.conduct(dataGrainByDefine, operatorFactory.get(DefaultOperators.INSERT));

1.3. Aggregation of data grains

DataGrain.aggregate - Used to aggregate multiple data grains in sequence

See detail sample of DataGrainTest.aggregate

import guru.mikelue.jdut.datagrain.DataGrain;

// dg_1 - The object of DataGrain
// dg_2 - Another object of DataGrain

// The data of dg_1 with data of dg_2....
DataGrain aggregatedDataGrains = dg_1.aggregate(dg_2);

1.4. Data grain decoration

A decorator is a lambda to modify a DataRow, which is the internal data of a DataGrain.

1.4.1. Decorate data grain(decorator)

You could implement functional interface of DataGrainDecorator to decorate an instance of DataGrain.

import guru.mikelue.jdut.datagrain.DataGrain;
import guru.mikelue.jdut.decorate.DataGrainDecorator;

DataGrainDecorator dataGrainDecorate = rowBuilder -> rowBuilder
	.field("col_1", Types.INTEGER, 55)
	.field("col_2", rowBuilder<Integer>.getData("col_1").orElse(0) + 1)
	.field("col_3", () -> "EXP-01");

DataGrain decoratedDataGrain = dataGrain.decorate(dataGrainDecorate);

1.4.2. Chaining of decorator

You could use method of DataGrainDecoration.chain() to chain multiple decorations.

import guru.mikelue.jdut.decorate.DataGrainDecorator;

DataGrainDecorator dataGrainDecorate_1 = (rowBuilder) -> {
	/* Your decoration...*/
}
DataGrainDecorator dataGrainDecorate_2 = (rowBuilder) -> {
	/* Your decoration...*/
}

DataGrainDecoration chainedDecorate = dataGrainDecorate_1.chain(dataGrainDecorate_2);

1.5. Data operator

You may use DataGrainOperator to implement your own operations to database.

DataRowsOperator

Accepts a list of DataRows

DataRowOperator

Accepts a DataRow

1.5.1. Composition of operations

You could composite operations for interception of data operation.

A SurroundOperator is a lambda to surround a DataGrainOperator. The returned object is another DataGrainOperator with same signature.

import guru.mikelue.jdut.operation.DefaultOperators;
import guru.mikelue.jdut.operation.DataGrainOperator;

DataGrainOperator specialInsert = DefaultOperators.INSERT.surroundedBy(
	surroundedOp -> (conn, dataGrain) -> {
		/* Your surrounding before the calling of surrounded operation */

		surroundedOp.operate(conn, dataGrain)

		/* Your surrounding after the calling of surrounded operation */
	};
)

1.5.2. Operate data in transaction

/**
 * By DataSourceConsumer
 */
import guru.mikelue.jdut.function.DatabaseTransactional;
import guru.mikelue.jdut.operation.DefaultOperators;

dataConductor.conduct(
	DefaultOperators.INSERT.surroundedBy(
		DatabaseTransactional::simple
	)
);

1.6. Data types

1.6.1. For Java API

The build-in operators of JDUT would use the type of value to decide which method of setXXX in PreparedStatement to be called for setting parameter of SQL.

java.util.Date

the engine would use PreparedStatement.setTimestamp().

Type cannot be decided

the engine would use PreparedStatement.setObject()

1.6.2. For YAML

  1. The engine(loading YAML) builds the data into DataGrain.
    Hence the target type is the decided by the conversion of YAML engine.

  2. You may assign desired type of data by YAML tagging

%TAG !jdut! tag:jdut.mikelue.guru:1.0/
%TAG !dbtype! tag:jdut.mikelue.guru:jdbcType:1.8/
%TAG !sql! tag:jdut.mikelue.guru:sql:1.0/

'''
- !sql!table tab_1 [
  # With column name
  {
	  col_1: !dbtype!smallint 10, col_2: "String Value",
	  col_3: !dbtype!timestamp "2010-05-05 10:20:35+08"
  },
  # Other rows...
]

1.6.3. Scalar Value

YAML Type : JDBC type Accept type of JDBC Misc

!!binary : VARBINARY

BINARY, BLOB, LONGVARBINARY, VARBINARY

*1 as base64

!!bool : BOOLEAN

BIT, BOOLEAN,
TINYINT, SMALLINT, INTEGER, BIGINT,
DECIMAL, NUMERIC, REAL,
DOUBLE, FLOAT

If the target type of JDBC is numeric: True(1), False(0)

!!float : DOUBLE

DECIMAL, DOUBLE, FLOAT, REAL, NUMERIC

*1

!!int : INTEGER

TINYINT, SMALLINT, INTEGER, BIGINT,
NUMERIC, DECIMAL, REAL,
DOUBLE, FLOAT

*1

!!str : VARCHAR

CHAR, VARCHAR, LONGVARCHAR,
NCHAR, NVARCHAR, LONGNVARCHAR,
CLOB, NCLOB

!!timestamp : TIMESTAMP

DATE, TIME, TIMESTAMP

*1

!!null : null value

doesn’t matter

1* - Could be converted to text-type of SQL

NUMERIC and DECIMAL

the value would be converted into BigDecimal.

1.6.4. Complex Value

For type of complex value, you could use Supplier<?> to create data of these types: ARRAY, DISTINCT, JAVA_OBJECT, OTHER, REF, REF_CURSOR, ROWID, STRUCT

2. YAML API

The configuration of YAML condcutor:

ConductorConfig

Bean as facade of configuration for conducting data.
The practices of the config are defined by implementation of engine.
The YamlConductorFactory.build() or YamlConductorFactory.conductResource() method let you set-up the instance of ConductorConfig.

ConductorConfig.Builder

The fed object of Consumer to set-up an instance of ConductorConfig

2.1. Set-up YamlConductorFactory

YamlConductorFactory

The most important service to build an instance of DuetConductor.
This factory depends on DataSource given to build() method.
You can define ConductorConfig to customize DataGrainDecorator or JdbcFunction, etc,.

DefaultOperatorFactory

The default factory used by YamlConductorFactory.
You may customize the factory by using ConductorConfig.Builder.operatorFactory() to set the instance.

// dataSource - The initialized data source

YamlConductorFactory yamlConductor = YamlConductorFactory.build(
	dataSource,
	builder -> builder
		// The factory of operator
		.operatorFactory(defaultOperationFactory)
		// The optional file loader for convertion string of file name to InputStream or Reader
		.resourceLoader((fileName) -> new FileReader(fileName))
		.namedOperator(
			"INSERT_AND_CHECK",
			(connection, dataRows) -> { /* Your database operations */ }
		)
		.namedDecorator(
			"[V1]",
			(dataRowBuilder) -> { /* Your data decoration */ }
		)
		.namedJdbcFunction(
			"func_1",
			connection -> { /* Your code of JDBC */ }
		)
);

2.2. Build DuetConductor

DuetConductor

This interface defines build/clean data for unit test.

import guru.mikelue.jdut.DuetConductor;

DuetConductor testConductor_1 = yamlFactory.conductNamedResource(
	"org/your/package/CarDaoTest-addNew.yaml",
	/**
	 * You could overrides configuration of YamlConductorFactory
	 */
	configure -> configure
		.namedDecorator("add_value", your_decorator)
	// :~)
);

// Consturct a content of YAML with Yaml.DEFAULT_TAGS directly(instead of file)
DuetConductor testConductor_2 = yamlConductor.conduct(
   YamlTags.DEFAULT_TAGS +
   "---\n" +
   "- !sql!table tab_1: {" +
   "  col_1: 40, col_2: \"VGA-1\"" +
   "}"
);

2.2.1. Resource loading

While using YamlConductorFactory.conductResource(String, Consuemr<ConductorConfig.Builder>) to load YAML file,+ you could configure how to convert a String to a Reader by ConductorConfig.Builder.resourceLoader().

See ReaderFunctions for build-in functions.


2.2.2. Execute build/clean

The DuetConductor should be used in proper event defined by testing framework.

For JUnit4
For TestNG

Following example to execute the two methods when testing:

testConductor_1.build();
testConductor_2.build();

try {
	/* Your testing code */
} finally {
	testConductor_2.clean();
	testConductor_1.clean();
}

3. Operators/Decorators

3.1. Build-in operators

The DefaultOperators constructs operator-inspection of vendor-specific, uses it would be a good idea.

  • INSERT

    • As DefaultOperators::insert

    • Use of SQL INSERT <table_name>, directly.

  • UPDATE

    • As DefaultOperators::update

    • Use of SQL UPDATE <table_name> SET <col_1> = <v_1>, …​ WHERE <conditions>, directly

  • REFRESH

    • As DefaultOperators::refresh

    • Gets the data, if the data is existing, then updates the data

    • Otherwise, inserts the data

  • DELETE

    • As DefaultOperators::delete

    • Use of SQL DELETE FROM <table_name> WHERE <conditions>, directly.

  • DELETE_ALL

    • As DefaultOperators::deleteAll

    • Use of SQL DELETE FROM <table_name>, directly.

  • TRUNCATE

    • As DefaultOperators::truncate

    • Use of SQL TRUNCATE TABLE <table_name>, directly.

  • NONE

    • As DefaultOperators::none

    • This operation has no effect to database.

import guru.mikelue.jdut.operation.DefaultOperatorFactory;
import guru.mikelue.jdut.operation.DefaultOperators;

DefaultOperatorFactory factory = DefaultDataOperatorFactory.build(
	dataSource, builder -> {}
);

dataConductor.conduct(
	factory.get(DefaultOperators.INSERT),
	dataGrain
);

3.2. Build-in decorators

3.2.1. Value replacement

import guru.mikelue.jdut.decorate.ReplaceFieldDataDecorator;
import guru.mikelue.jdut.decorate.DataGrainDecorator;

/**
 * The decorate to replace string by assigned value
 */
DataGrainDecorator replaceDecorate = ReplaceFieldDataDecorator.build(
	builder -> builder
		.replaceWith(30, 90); // Replacement by object
		.replaceWith("[V1]", () -> 10); // Replacement by supplier
		.replaceWith(dataField -> dataField.getColumnName().equals("col_9"), "Another Value"); // Replacement by predict of datafield
);
Default value

You can replace value with null-value checking

  • DataFieldPredicates.nullValue(tableName, columnName) - Builds a predicate to check null value for a column on table

  • DataFieldPredicates::nonSupplier - A predicate to check if the data of field doesn’t comes from Supplier

  • This is used if you don’t want to load data before the processing of operator

// builder - builder of ReplaceFieldDataDecorator.build()

builder -> builder
	.replaceWith(DataFieldPredicates.nullValue("tab_1", "col_1"), "DEFAULT-VALUE")

3.2.2. Not defined column

You may use DataRowBuilderPredicates::notExistingColumn to predicate a decorator {a row}[DataRow] with not-defined column.

import guru.mikelue.jdut.function.DataRowBuilderPredicates;
import guru.mikelue.jdut.decorate.DataGrainDecorator;

DataGrainDecorator decorator = (rowBuilder -> rowBuilder.field("col_1", -1))
	.predicate(DataRowBuilderPredicates.notExistingColumn("tabl_1", "col_1"));

3.3. Conductor Context

While the DataConductor is executing, you may access its current working object of Connection provided from ConductorContext, which keeps the connection in ThreadLocal.

A good example to use conductor context is value function, you may access current session of database while generating data of a field.

Lazy loading of value function

Since conductor context is available only when it is executing,
you should keep in mind that DataGrainDecorator shouldn’t access DataRow.Builder.getData(<field_name>) to trigger the value function.
Instead, using DataRow.Builder.getDataSupplier(<field_name>).isPresent() to check whether or not the value comes from a value function.

You must not close the object of Connection from the context.

Connection conn = ConductorContext.getCurrentConnection()
	.orElseThrow(() -> new Exception("NoContext"));

/* 1) Use of connection */
/* 2) Do not close it!! */

3.3.1. Threading

Any execution of DataConductor.conduct(…​) would initialize a new object of Connection, which is put into ConductorContext.


4. Database Vendor

4.1. Vendor-specific operators

DefaultOperatorFactory would try to fetch best-implementation for vendor-specific operations.

You can override the operators implemented by JDUT.

The method of DefaultOperatorFactory.Builder.add accepts two arguments:

OperatorPredicate

The method of testMetaData(DatabaseMetaData) is used to check the meta data of current connection to database.
This predicate returning true value means to use your mapping of operators.

Map<String, DataGrainOperator>

The mapping is used to fetch operator by name for the fisrt matched OperatorPredicate.

You don’t have to define all of the names defined in DefaultOperators in the Map of operators. DefaultOperatorFactory would fetch a default one if your mapping doesn’t contain the name.

The first matched predicte would be selected for the vendor-specific operation

import guru.mikelue.jdut.operation.DefaultOperatorFactory;
import guru.mikelue.jdut.operation.OperatorFactory;

// dataSource - The initialized data source
// yourMapOfOperators - Your customized mapping of operators
OperatorFactory yourFactory = DefaultOperatorFactory.build(
	dataSource,
	builder -> builder
		.add(
			metaData -> metaData.getDriverName().contains("something"),
			yourMapOfOperators
		)
);

4.2. Notes

Following list states the databases and JDBC drivers which have been tested:

Database Vendor Database Version JDBC Version Misc

PostgreSql

9.4

jdbc41

MySql

5.6

5.1

Oracle

11g express

11.2.0(thin)

MsSql

2013 express

4.2

H2

1.4

1.4

HsqlDb

2.3

2.3

Derby

10.11

10.11

4.2.1. Keys of row

In order to execute DELETE or UPDATE more precisely, following priority for looking up one or multiple columns in a table is applied to build WHERE statement:

  1. Setting of keys in SchemaTable.Builder.keys(String…​) or keys of YAML

  2. primary key

  3. unique index with least number of columns and all of their value have non-null value