---
title: Transformation Primitives
description: Pipeline transformation task reference
weight: 20
---

Transformation tasks modify the fields of each data entry.

Syntax:

    - TASKNAME: ["[!]CONDITION", "opt1", "opt2", ...]

Use `''` as condition to always execute. Use `!COND` to execute only when condition is False.


## Quick reference

```yaml
    - field_noop:         [COND]                                : do nothing
    - field_toint:        [COND, field1, ...]                   : convert fields to int
    - field_tofloat:      [COND, field1, ...]                   : convert fields to float
    - field_tostring:     [COND, field1, ...]                   : convert to string
    - field_nospace:      [COND, field1, ...]                   : remove all whitespaces from fields
    - field_regexp_sub:   [COND, field, 'pattern', 'replace']   : apply regexp/replace
    - field_set:          [COND, field, 'value']                : create field with value
    - field_copy:         [COND, field1, field2]                : field1 to field2
    - field_rename:       [COND, field1, field2]                : field1 to field2
    - field_merge:        [COND, field1, field2, field3]        : field1 + field2 > field3
    - field_delete:       [COND, field1, field2, ...]           : remove fields
    - field_keep:         [COND, field1, field2, ...]           : keep only these fields (and classname/keyname)
    - field_lower:        [COND, field1, ...]                   : lowercase
    - field_upper:        [COND, field1, ...]                   : uppercase
    - field_date_now:     [COND, field1, ...]                   : set field(s) to YYYY-MM-DD
    - field_time_now:     [COND, field1, ...]                   : set field(s) to HH:MM:SS
    - field_datetime_now: [COND, field1, ...]                   : set field(s) to YYYY-MM-DD HH:MM:SS
    - field_uuid:         [COND, field1, ...]                   : set field(s) to random UUID string (distinct)
    - field_append:       [COND, field, 'suffix']               : append a suffix to field
    - field_prepend:      [COND, field, 'prefix']               : prefix field with prefix string
    - field_md5:          [COND, dst, src1, src2, ...]          : dst = md5(concat(src1, src2, ...))
    - field_join:         [COND, sep, dst_field, src1, src2, ...]: join fields with separator
    - field_datetime:     [COND, target, source, delta]         : compute datetime into target field
    - field_truncate:     [COND, field, length]                 : cut field content to length chars
    - field_lookup:       [COND, dstfield, schemaname, srcfield]: lookup entry by keyname, set dstfield or ""
    - align_subnet4:      [COND, field]                         : align IPv4/mask to subnet boundaries
    - discard:            [COND]                                : eliminate full entry
    - exit:               [COND, 'status']                      : stop processing tasks, keep entry
```


## field_noop

    tasks:
    - field_noop: [CONDITION]

Performs nothing. Useful as a placeholder or for testing conditions.


## field_set

    tasks:
    - field_set: [CONDITION, "fieldname", "AnyValue"]

Creates or overwrites `fieldname` and sets its value.


## field_copy

    tasks:
    - field_copy: [CONDITION, "field1", "field2"]

Creates/overwrites `field2` with the value of `field1`.


## field_rename

    tasks:
    - field_rename: [CONDITION, "field1", "field2"]

Renames `field1` to `field2`. Creates/overwrites `field2` with `field1`'s value and removes `field1`.


## field_delete

    tasks:
    - field_delete: [CONDITION, "field1", "field2", ...]

Removes all specified fields.


## field_keep

    tasks:
    - field_keep: [CONDITION, "field1", "field2", ...]

Keeps provided fields **only**. Removes all other fields (except `classname` and `keyname`).


## field_lower

    tasks:
    - field_lower: [CONDITION, "field", "field", ...]

Converts field value(s) to lowercase.


## field_upper

    tasks:
    - field_upper: [CONDITION, "field", "field", ...]

Converts field value(s) to uppercase.


## field_toint

    tasks:
    - field_toint: [CONDITION, "field", "field", ...]

Converts a field's value to integer.


## field_tofloat

    tasks:
    - field_tofloat: [CONDITION, "field", "field", ...]

Converts a field's value to floating point value.


## field_tostring

    tasks:
    - field_tostring: [CONDITION, "field", "field", ...]

Converts a field's value to a string.


## field_nospace

    tasks:
    - field_nospace: [CONDITION, "field", "field", ...]

Removes all whitespace and tab characters from field values. Faster than a general regexp task.


## field_regexp_sub

    tasks:
    - field_regexp_sub: [CONDITION, "fieldname", "pattern", "replace"]

Alters `fieldname`'s value by replacing a regexp pattern with the provided replacement. Accepts any Python standard regexp.

Example:

    tasks:
    - field_regexp_sub: ['', 'field_a', 'test', 'QWERTY']

    # Before : {'field_a': 'This is a test from unittest !'}
    # After  : {'field_a': 'This is a QWERTY from unitQWERTY !'}


## field_merge

    tasks:
    - field_merge: [CONDITION, 'field1', 'field2', 'field3']

Concatenates `field1` and `field2` into new `field3` (overwrites if already exists).

If field values are numerical, a mathematical addition is performed. If they are strings, string concatenation is performed.

Use `field_toint` / `field_tofloat` / `field_tostring` to handle type conversions beforehand.


## field_prepend

    tasks:
    - field_prepend: [CONDITION, 'field1', 'prefix']

Puts the `prefix` string in front of `field1`'s value.


## field_append

    tasks:
    - field_append: [CONDITION, 'field1', 'suffix']

Puts the `suffix` string at the end of `field1`'s value.


## field_date_now

    tasks:
    - field_date_now: [CONDITION, "field", "field", ...]

Sets field(s) to the current date in `YYYY-MM-DD` format.

Useful for automated or periodic imports to track in-sync/out-of-sync objects.


## field_time_now

    tasks:
    - field_time_now: [CONDITION, "field", "field", ...]

Sets field(s) to the current time in `HH:MM:SS` format.


## field_datetime_now

    tasks:
    - field_datetime_now: [CONDITION, "field", "field", ...]

Sets field(s) to the current datetime in `YYYY-MM-DD HH:MM:SS` format.


## field_uuid

    tasks:
    - field_uuid: [CONDITION, "field", "field", ...]

Creates/overwrites field(s) with a random UUID string. Each field receives a distinct UUID.

Use `field_copy` if the same UUID is needed in multiple fields.

See `field_md5` for a deterministic primary key.


## field_md5

new v3.24

    tasks:
    - field_md5: [CONDITION, dst, src1, src2, ...]

Computes md5 of the concatenated values of `src*` fields and stores the result in `dst`.

If any source field is missing, no md5 is computed and `dst` is not created.

Useful for creating a deterministic primary key by combining multiple fields.


## field_join

new v3.24

    tasks:
    - field_join: [CONDITION, separator, dst_field, src1, src2, src3, ...]

Joins multiple source fields into a destination field using a separator (similar to Python's `str.join()`). All values are converted to strings.

Example:

    tasks:
    - field_join: ['', '-', 'result', 'field1', 'field2']

    # Before : {'field1': 'hello', 'field2': 'world'}
    # After  : {'field1': 'hello', 'field2': 'world', 'result': 'hello-world'}

    - field_join: ['', ' ', 'fullname', 'firstname', 'lastname']

    # Before : {'firstname': 'John', 'lastname': 'Doe'}
    # After  : {'firstname': 'John', 'lastname': 'Doe', 'fullname': 'John Doe'}


## field_datetime

new v3.33

    tasks:
    - field_datetime: [CONDITION, target_field, source, delta]

Computes a datetime value and stores it in `target_field`.

**source** (required): one of:
- `now()` — current local datetime
- a field name present in the entry — its value is parsed as a datetime
- a datetime string in ISO format `YYYY-MM-DD HH:MM:SS`

**delta** (optional): a duration to add or subtract, in the form `[+-]<int><unit>`.

Supported units: `second`, `minute`, `hour`, `day`, `week`, `month`, `year` (plural form also accepted).

`delta` can also be the name of a field whose value contains the delta string.

Output format: `YYYY-MM-DD HH:MM:SS`

If source cannot be resolved or parsed, the target field is not set.

Examples:

    tasks:
    # set result to current datetime
    - field_datetime: ['', result, 'now()']

    # set expiry to start + 30 days
    - field_datetime: ['', expiry, start, '+30day']

    # subtract 3 months from a fixed date
    - field_datetime: ['', result, '2025-06-01 00:00:00', '-3month']

    # use a field as both source and delta
    - field_datetime: ['', result, last_sync, retention_delay]


## field_truncate

    tasks:
    - field_truncate: [CONDITION, field, length]

Cuts `field` content to `length` characters. The field value is converted to string first. If `length` is greater than the field length, the value is unchanged.

Example:

    tasks:
    - field_truncate: ['', description, 100]

    # Before : {'description': 'A very long description that exceeds the limit...'}
    # After  : {'description': 'A very long description that exceeds the limit...'}[0:100]


## field_lookup

new v3.33

    tasks:
    - field_lookup: [CONDITION, dstfield, schema_name, srcfield]

Looks up an entry in `schema_name` using the value of `srcfield` as the keyname. Sets `dstfield` to the keyname of the found entry, or to `""` if no match is found.

**schema_name** can be:
- `_user` — looks up a user by `login`
- `_group` — looks up a group by `keyname`
- `_role` — looks up a role by `keyname`
- `_permission` — looks up a permission by `keyname`
- any other schema name — looks up a `DataInstance` by `keyname`

Useful for validating references and resolving canonical keynames from external data.

Examples:

    tasks:
    # Check that a username from an import exists as a Cavaliba user
    - field_lookup: ['', resolved_user, _user, raw_login]

    # Before : {'raw_login': 'alice'}
    # After  : {'raw_login': 'alice', 'resolved_user': 'alice'}   (if alice exists)
    # After  : {'raw_login': 'nobody', 'resolved_user': ''}       (if not found)

    # Resolve a server from an external dataset
    - field_lookup: ['', server, myserver_schema, server_name]


## align_subnet4

new v3.24

    tasks:
    - align_subnet4: [CONDITION, fieldname]

Aligns an IPv4 address/mask to subnet boundaries. Transforms an IP address with prefix length to the network address with the same prefix length.

Example:

    tasks:
    - align_subnet4: ['', subnet_field]

    # Before : {'subnet_field': '10.1.1.1/24'}
    # After  : {'subnet_field': '10.1.1.0/24'}

    # Before : {'subnet_field': '192.168.50.100/16'}
    # After  : {'subnet_field': '192.168.0.0/16'}


## discard

    tasks:
    - discard: [CONDITION]

Eliminates the full entry. The entry is not written to the database.


## exit

new v3.33

    tasks:
    - exit: [CONDITION, status]

Stops processing remaining tasks for the current entry. The entry is kept (not discarded).

The optional `status` parameter sets the return value for other processing steps. Defaults to `'exit'` if omitted.

Example:

    tasks:
    - set_condition: [IS_DONE, field_match, status, '^done$']
    - exit: [IS_DONE, "all-done"]
    - field_set: ['', needs_processing, 'yes']

    # entries with status='done' skip remaining tasks and are saved as-is
