Persistence
Automatiko by default runs without any persistent storage so it can be used without additional components. Though in many cases workflow instances are long lived and thus require storage to persist its state.
Automatiko comes with several options to chose from
-
File system based storage
-
Database based storage
-
Amazon DynamoDB based storage
-
Apache Cassandra based storage
| More is on their way and as soon as they will be implemented they will be listed here. |
File system based storage
The most basic but quite powerful storage option is based on file system. This means each workflow instance will be stored in dedicated file in configured directory.
In addition to file itself, Automatiko will store some metadata that allows to more efficiently look up instances without loading it completely.
Depending on file system capabilities these metadata will be stored as
-
extended file attributes if file system supports it
-
dot filenext to the actual file as a fallback option when file system does not support extended attributes
Configuration
To use file system based perssitence your service must have following dependency
<dependency>
<groupId>io.automatiko.addons</groupId>
<artifactId>automatiko-filesystem-persistence-addon</artifactId>
</dependency>| Note that each persistence addon comes with job service implementation as well to provide persistence of timers defined in workflows |
Additional configuration should be placed in application.properties
| Property name | Environment variable | Description | Required | Default value | BuildTime only |
|---|---|---|---|---|---|
quarkus.automatiko.persistence.type |
Specify what persistence should be used |
No |
Yes |
||
quarkus.automatiko.persistence.filesystem.path |
QUARKUS_AUTOMATIKO_PERSISTENCE_FILESYSTEM_PATH |
Location on file system that will be used to store persistent state |
Yes |
No |
|
quarkus.automatiko.jobs.type |
Specifies type of jobs implementation to be used |
No |
Yes |
||
quarkus.automatiko.jobs.filesystem.path |
QUARKUS_AUTOMATIKO_JOBS_FILESYSTEM_PATH |
Location on file system where jobs persistent state will be stored |
Yes |
No |
|
quarkus.automatiko.jobs.filesystem.threads |
QUARKUS_AUTOMATIKO_JOBS_FILESYSTEM_THREADS |
Specifies how many threads should be used for jobs execution |
No |
1 |
No |
| This storage option can be used for pretty much any use case but it has certain drawbacks depending on the file system being used. It might not be the best option when horizontal scalability is important. |
Database based storage
Database based storage is mainly dedicated to cover Database record processing
as it requires workflow data objects to be
-
simple types
-
entities
-
collection of entities
Its main applicability is to store workflow state next to the data it is processing.
| Data model of the workflow becomes an entity as well, which on database level will be represented as table. |
Every workflow definition will be stored in dedicated table that is constructed based on workflow identifier and version. The table will also have all simple type data objects in the same table and reference any entity data objects.
Green tables are those defined as data objects and annotated as entities. Orange tables are generated based on workflow definitions
-
vacations is the top level (public) process for handling vacation requests
-
requests is the sub workflow (private) that handles individual request
Blue table is used by jobs service that stores its data in db as well to support persistent timers defined in the workflow.
Data management
Since workflows operate on database records (entities defined as data objects) it has an option to manage these records. One of the important aspects is what should happen to the data after workflow instance completes?
There are two options provided by Automatiko
-
keep the data as they are at workflow instance completion
-
remove data that the workflow instance has worked with
| By default data is not removed at the completion of workflow instance |
Configuration
To use database based persistence your service must have following dependency
<dependency>
<groupId>io.automatiko.addons</groupId>
<artifactId>automatiko-db-persistence-addon</artifactId>
</dependency>| Note that each persistence addon comes with job service implementation as well to provide persistence of timers defined in workflows |
Additional configuration should be placed in application.properties
| Property name | Environment variable | Description | Required | Default value | BuildTime only |
|---|---|---|---|---|---|
quarkus.automatiko.persistence.type |
Specify what persistence should be used |
No |
Yes |
||
quarkus.automatiko.persistence.db.remove-at-completion |
Specifies if entities created during instance execution should be removed when instance completes |
No |
false |
Yes |
|
quarkus.automatiko.jobs.db.interval |
QUARKUS_AUTOMATIKO_JOBS_DB_INTERVAL |
Specifies interval (in minutes) how often look for another chunk of jobs to execute |
No |
60 |
No |
quarkus.automatiko.jobs.db.threads |
QUARKUS_AUTOMATIKO_JOBS_DB_THREADS |
Specifies how many threads should be used for job execution |
No |
1 |
No |
| Property name | Environment variable | Description | Required | Default value | BuildTime only |
|---|---|---|---|---|---|
quarkus.datasource.db-kind |
Specify what kind of data base should be used |
Yes |
Yes |
||
quarkus.datasource.username |
QUARKUS_DATASOURCE_USERNAME |
Specify user name to be used to connect to data base |
Yes |
No |
|
quarkus.datasource.password |
QUARKUS_DATASOURCE_PASSWORD |
Specify password to be used to connect to data base |
Yes |
No |
|
quarkus.datasource.jdbc.url |
QUARKUS_DATASOURCE_JDBC_URL |
Specify url to be used to connect to data base |
Yes |
No |
Full configuration reference for data source can be found here
DynamoDB based storage
DynamoDB based storage allows to rely on Amazon managed DynamoDB. It allows to offload most of the storage heavy lifting (such as replication, scalability etc) to managed persistence service.
Automatiko takes advantages of highly scalable nature of DynamoDB that nicely fits into workflow data model - being key value based.
Every workflow definition will be stored in dedicated table that is constructed based on workflow identifier and version. That will result in each workflow definition to have dedicated storage location. Automatiko by default creates all required tables so there is no need to dive into the details but if such need arise automatic table creation can be disabled.
The table structure will have following attributes
-
InstanceId - (of type string) this is the unique id of each workflow instance
-
Content - (of type binary) this is the actual workflow instance serialized
-
Tags - (of type list) this is the workflow instance tags - it also includes
instance idandbusiness key
| In addition when table is created it assigns default values for read and write capacity set to 10, it can be changed via configuration properties |
Configuration
To use DynamoDB based persistence your service must have following dependency
<dependency>
<groupId>io.automatiko.addons</groupId>
<artifactId>automatiko-dynamodb-persistence-addon</artifactId>
</dependency>| Note that each persistence addon comes with job service implementation as well to provide persistence of timers defined in workflows |
Additional configuration should be placed in application.properties
| Property name | Environment variable | Description | Required | Default value | BuildTime only |
|---|---|---|---|---|---|
quarkus.automatiko.persistence.type |
Specify what persistence should be used |
No |
Yes |
||
quarkus.automatiko.persistence.dynamodb.create-tables |
QUARKUS_AUTOMATIKO_PERSISTENCE_DYNAMODB_CREATE_TABLES |
Specifies if tables should be automatically created |
No |
true |
No |
quarkus.automatiko.persistence.dynamodb.read-capacity |
QUARKUS_AUTOMATIKO_PERSISTENCE_DYNAMODB_READ_CAPACITY |
Specifies read capacity to be applied to created DynamoDB tables |
No |
10 |
No |
quarkus.automatiko.persistence.dynamodb.write-capacity |
QUARKUS_AUTOMATIKO_PERSISTENCE_DYNAMODB_WRITE_CAPACITY |
Specifies write capacity to be applied to created DynamoDB tables |
No |
10 |
No |
quarkus.automatiko.jobs.dynamodb.create-tables |
QUARKUS_AUTOMATIKO_JOBS_DYNAMODB_CREATE_TABLES |
Specifies if tables should be automatically created |
No |
true |
No |
quarkus.automatiko.jobs.dynamodb.read-capacity |
QUARKUS_AUTOMATIKO_JOBS_DYNAMODB_READ_CAPACITY |
Specifies read capacity to be applied to created DynamoDB tables |
No |
10 |
No |
quarkus.automatiko.jobs.dynamodb.write-capacity |
QUARKUS_AUTOMATIKO_JOBS_DYNAMODB_WRITE_CAPACITY |
Specifies write capacity to be applied to created DynamoDB tables |
No |
10 |
No |
quarkus.automatiko.jobs.dynamodb.interval |
QUARKUS_AUTOMATIKO_JOBS_DYNAMODB_INTERVAL |
Specifies interval (in minutes) how often look for another chunk of jobs to execute |
No |
60 |
No |
quarkus.automatiko.jobs.dynamodb.threads |
QUARKUS_AUTOMATIKO_JOBS_DYNAMODB_THREADS |
Specifies how many threads should be used for job execution |
No |
1 |
No |
| Property name | Environment variable | Description | Required | Default value | BuildTime only |
|---|---|---|---|---|---|
quarkus.dynamodb.endpoint-override |
QUARKUS_DYNAMODB_ENDPOINT_OVERRIDE |
The endpoint URI with which the SDK should communicate |
Yes |
No |
|
quarkus.dynamodb.aws.region |
QUARKUS_DYNAMODB_AWS_REGION |
An Amazon Web Services region that hosts the given service |
Yes |
No |
|
quarkus.dynamodb.aws.credentials.type |
QUARKUS_DYNAMODB_AWS_CREDENTIALS_TYPE |
Configure the credentials provider that should be used to authenticate with AWS |
Yes |
No |
Full configuration reference for dynamodb can be found here
Apache Cassandra based storage
Apache Casandra based storage allows to rely on advanced distributed data store for highly available use cases. It enables use of multiple data centers to replicate workflow instance data securely to make it available in all available locations.
Every workflow definition will be stored in dedicated table that is constructed based on workflow identifier and version. That will result in each workflow definition to have dedicated storage location. Automatiko by default creates all required tables so there is no need to dive into the details but if such need arise automatic table creation can be disabled.
The table structure will have following attributes
-
InstanceId - (of type text) this is the unique id of each workflow instance
-
Content - (of type blob) this is the actual workflow instance serialized
-
Tags - (of type set) this is the workflow instance tags - it also includes
instance idandbusiness key
In addition to tables, Automatiko will create key space as well with configured name. If name is not given then it
defaults to automatiko.
Automatic keyspace creation is most basic setup as it uses simple replication strategy with replication
factor set to 1. For more advanced (production like) use cases key space should be created manually and its name
should be given via configuration.
|
Next to tables, each workflow definition will get an index to be able to query more efficiently.
Configuration
To use Apache Cassandra based persistence your service must have following dependency
<dependency>
<groupId>io.automatiko.addons</groupId>
<artifactId>automatiko-cassandra-persistence-addon</artifactId>
</dependency>| Note that each persistence addon comes with job service implementation as well to provide persistence of timers defined in workflows |
Additional configuration should be placed in application.properties
| Property name | Environment variable | Description | Required | Default value | BuildTime only |
|---|---|---|---|---|---|
quarkus.automatiko.persistence.type |
Specify what persistence should be used |
No |
Yes |
||
quarkus.automatiko.persistence.cassandra.create-keyspace |
QUARKUS_AUTOMATIKO_PERSISTENCE_CASSANDRA_CREATE_KEYSPACE |
Specifies if keyspace should be automatically created |
No |
true |
No |
quarkus.automatiko.persistence.cassandra.create-tables |
QUARKUS_AUTOMATIKO_PERSISTENCE_CASSANDRA_CREATE_TABLES |
Specifies if tables should be automatically created |
No |
true |
No |
quarkus.automatiko.persistence.cassandra.keyspace |
QUARKUS_AUTOMATIKO_PERSISTENCE_CASSANDRA_KEYSPACE |
Specifies key space name to be used for tables |
No |
10 |
No |
quarkus.automatiko.jobs.cassandra.create-keyspace |
QUARKUS_AUTOMATIKO_JOBS_CASSANDRA_CREATE_KEYSPACE |
Specifies if keyspace should be automatically created |
No |
true |
No |
quarkus.automatiko.jobs.cassandra.create-tables |
QUARKUS_AUTOMATIKO_JOBS_CASSANDRA_CREATE_TABLES |
Specifies if tables should be automatically created |
No |
true |
No |
quarkus.automatiko.jobs.cassandra.keyspace |
QUARKUS_AUTOMATIKO_JOBS_CASSANDRA_KEYSPACE |
Specifies key space name to be used for tables |
No |
10 |
No |
quarkus.automatiko.jobs.cassandra.interval |
QUARKUS_AUTOMATIKO_JOBS_CASSANDRA_INTERVAL |
Specifies interval (in minutes) how often look for another chunk of jobs to execute |
No |
60 |
No |
quarkus.automatiko.jobs.cassandra.threads |
QUARKUS_AUTOMATIKO_JOBS_CASSANDRA_THREADS |
Specifies how many threads should be used for job execution |
No |
1 |
No |
| Property name | Environment variable | Description | Required | Default value | BuildTime only |
|---|---|---|---|---|---|
quarkus.cassandra.contact-points |
QUARKUS_CASSANDRA_CONTACT_POINTS |
Specify url to the cassandra data base |
Yes |
No |
|
quarkus.cassandra.local-datacenter |
QUARKUS_CASSANDRA_LOCAL_DATACENTER |
Specify data senter to be used on cassandra |
Yes |
No |
Full configuration reference for Apache Cassandra can be found here