Using arrays, strings, and complex structures with DataPrime

Goal

By the end of this guide you should be able to manipulate arrays, transform and analyze strings, and access deeply nested fields. You’ll learn how to split strings, decode values, flatten complex objects, and troubleshoot keypath issues that can break queries.

Why it matters

Real-world logs rarely come in clean. Arrays often need to be expanded, strings need parsing or decoding, and deeply nested or oddly named fields can get in the way of querying. This guide shows you how to handle that complexity with confidence using DataPrime.

explode – Split array elements into rows

Description

Use explode when you need to analyze each element in an array as its own document. This is useful for permissions, tags, error lists, and other multi-valued fields.

Syntax

explode <array_field> into <new_field> original preserve

Example – Flatten scopes for permission analysis

Input data

{ "user_id": "1", "scopes": ["read", "write"] }

Query

explode scopes into scope original preserve

Result

{ user_id: 1, scope: read, scopes: [read, write] }
{ user_id: 1, scope: write, scopes: [read, write] }

arrayConcat – Combine multiple arrays into one

Description

Use arrayConcat to merge two or more arrays into a single array field. Ideal for combining values split across fields (e.g., job queues, error types).

Syntax

create <new_field> from arrayConcat(array1, array2, ...)

Example – Merge frontend and backend error arrays

Input data

{
  "frontend_errors": ["missing_token", "timeout"],
  "backend_errors": ["db_error"]
}

Query

create all_errors from arrayConcat(frontend_errors, backend_errors)

Result

{ all_errors: [missing_token, timeout, db_error] }

arrayAppend – Add a value to the end of an array

Description

Use arrayAppend to add a value to the end of an array field. This is useful when the value is available, but stored separately from the array.

Syntax

replace <array_field> with <array_field>.arrayAppend(<value>)

Example 1 – Add a static job step

Input data

{ "steps": ["extract", "transform"] }

Query

replace steps with steps.arrayAppend('finalize')

Result

{ steps: [extract, transform, finalize] }

Example 2 – Append a field value into the array

Input data

{
  "jobs": ["job-1", "job-2"],
  "extra_job": "job-3"
}

Query

replace jobs with jobs.arrayAppend(extra_job)

Result

{ jobs: [job-1, job-2, job-3] }

arrayContains – Check if a value exists in an array

Description

Use arrayContains to determine if a specific value appears inside an array. Returns a boolean.

Syntax

create <flag_field> from <array>.arrayContains(<value>)

Example – Flag blocked IP addresses

Input data

{
  "client_ip": "1.2.3.4",
  "blocked_ips": ["1.2.3.4", "5.6.7.8"]
}

Query

create is_blocked_ip from blocked_ips.arrayContains(client_ip)

Result

{ is_blocked_ip: true }

Parsing and transforming strings

arraySplit – Split a string into parts

Description

Use arraySplit to break a string into parts using a delimiter. Often used for names, paths, versions, or tags.

Syntax

<field>.arraySplit(<delimiter>)[<index>]

Example – Split full name into first and last

Input data

{ "name": "Joe-Nelson" }

Query

create first_name from name.arraySplit("-")[0]
| create last_name from name.arraySplit("-")[1]

Result

{ first_name: Joe, last_name: Nelson }

arrayJoin – Join array values into a string

Description

Use arrayJoin to convert an array into a readable string using a delimiter.

Syntax

<array_field>.arrayJoin(<delimiter>)

Example – Format a user action log

Input data

{ "users": ["Emma", "Ofri", "Zev"], "action": "LOGIN" }

Query

create msg from `Users {users.arrayJoin(', ')} performed action {action}`

Result

{ msg: Users Emma, Ofri, Zev performed action LOGIN }

urlDecode / urlEncode – Decode or encode URL-safe strings

Description

Use urlDecode to make encoded strings readable. Use urlEncode when you need to safely transmit or store text.

Syntax

<field>.urlDecode()
<field>.urlEncode()

Example – Decode a query string parameter

Input data

{ "query_string_parameters": { "name": "Tom%20Kosman" } }

Query

replace query_string_parameters.name with query_string_parameters.name.urlDecode()

Result

{ query_string_parameters: { name: Tom Kosman } }

decodeBase64 – Decode base64 strings

Description

Use decodeBase64 to convert encoded strings (e.g., compressed URLs or payloads) into readable values.

Syntax

<field>.decodeBase64()

Example – Decode a base64 URL

Input data

{ "full_url_encoded": "aHR0cHM6Ly9jb3JhbG9naXguY29t" }

Query

create full_url from full_url_encoded.decodeBase64()

Result

{ full_url: https://coralogix.com }

contains, startsWith – Check for string patterns

Description

Use contains or startsWith to detect substrings in a string field. Useful for filtering by prefix, domain, or label.

Syntax

<field>.contains(<substring>)
<field>.startsWith(<substring>)

Example – Identify log types or domains

Input data

{ "url": "https://google.com/search", "msg": "ERROR timeout exceeded" }

Query

filter url.contains("google.com")
| create is_error_log from msg.startsWith("ERROR")

Result

{ is_error_log: true }

choose – Flatten deeply nested fields

Description

Use choose to extract a deeply nested field and bring it to the top level. Makes it easier to read and query.

Syntax

choose <deep_field> as <alias>

Example – Simplify metric access

Input data

{
  "http_request": {
    "metrics": {
      "bytes_metrics": {
        "bytes_received": 1024
      }
    }
  }
}

Query

choose http_request.metrics.bytes_metrics.bytes_received as bytes_received

Result

{ bytes_received: 1024 }

extract ... using kv() – Parse key-value strings

Description

Use extract ... using kv() to convert a structured string into a map of fields. Add datatypes to cast values into numbers or timestamps.

Syntax

extract <field> into <object> using kv(pair_delimiter, key_delimiter) datatypes ...

Example – Parse and cast query parameters

Input data

{ "msg": "query_id=200&duration_ms=1000" }

Query

extract msg into data using kv(pair_delimiter='&', key_delimiter='=') datatypes duration_ms:number

Result

{ data: { query_id: 200, duration_ms: 1000 } }

Bracket notation – Access special keypaths

Description

Use bracket notation to access keys that contain dots, spaces, or special characters. Required in archive/compliance mode.

Syntax

<root_field>['key.with.dot.or.special-character']

Example – Filter by a key with dots

Input data

{ "k8s.tags.process_id": 1234 }

Query

filter k8s['tags.process_id'] == 1234

Result

{ k8s.tags.process_id: 1234 }

Common pitfalls or gotchas

• Exploding arrays removes other fields unless you preserve them. Always use original preserve unless you want a minimal result.
• String length limits can break string functions. If a field is longer than 256 characters, some functions may silently return null in high-tier mode.
• Always quote bracketed keypaths. Even one missed bracket or dot can break your query.