Using arrays, strings, and complex structures with DataPrime
Goal
By the end of this guide you should be able to manipulate arrays, transform and analyze strings, and access deeply nested fields. You’ll learn how to split strings, decode values, flatten complex objects, and troubleshoot keypath issues that can break queries.
Why it matters
Real-world logs rarely come in clean. Arrays often need to be expanded, strings need parsing or decoding, and deeply nested or oddly named fields can get in the way of querying. This guide shows you how to handle that complexity with confidence using DataPrime.
explode – Split array elements into rows
Description
Use explode when you need to analyze each element in an array as its own document. This is useful for permissions, tags, error lists, and other multi-valued fields.
Syntax
explode <array_field> into <new_field> original preserve
Example – Flatten scopes for permission analysis
Input data
{ "user_id": "1", "scopes": ["read", "write"] }
Query
explode scopes into scope original preserve
Result
{ user_id: 1, scope: read, scopes: [read, write] }
{ user_id: 1, scope: write, scopes: [read, write] }
arrayConcat – Combine multiple arrays into one
Description
Use arrayConcat to merge two or more arrays into a single array field. Ideal for combining values split across fields (e.g., job queues, error types).
Syntax
create <new_field> from arrayConcat(array1, array2, ...)
Example – Merge frontend and backend error arrays
Input data
{
"frontend_errors": ["missing_token", "timeout"],
"backend_errors": ["db_error"]
}
Query
create all_errors from arrayConcat(frontend_errors, backend_errors)
Result
{ all_errors: [missing_token, timeout, db_error] }
arrayAppend – Add a value to the end of an array
Description
Use arrayAppend to add a value to the end of an array field. This is useful when the value is available, but stored separately from the array.
Syntax
replace <array_field> with <array_field>.arrayAppend(<value>)
Example 1 – Add a static job step
Input data
{ "steps": ["extract", "transform"] }
Query
replace steps with steps.arrayAppend('finalize')
Result
{ steps: [extract, transform, finalize] }
Example 2 – Append a field value into the array
Input data
{
"jobs": ["job-1", "job-2"],
"extra_job": "job-3"
}
Query
replace jobs with jobs.arrayAppend(extra_job)
Result
{ jobs: [job-1, job-2, job-3] }
arrayContains – Check if a value exists in an array
Description
Use arrayContains to determine if a specific value appears inside an array. Returns a boolean.
Syntax
create <flag_field> from <array>.arrayContains(<value>)
Example – Flag blocked IP addresses
Input data
{
"client_ip": "1.2.3.4",
"blocked_ips": ["1.2.3.4", "5.6.7.8"]
}
Query
create is_blocked_ip from blocked_ips.arrayContains(client_ip)
Result
{ is_blocked_ip: true }
Parsing and transforming strings
arraySplit – Split a string into parts
Description
Use arraySplit to break a string into parts using a delimiter. Often used for names, paths, versions, or tags.
Syntax
<field>.arraySplit(<delimiter>)[<index>]
Example – Split full name into first and last
Input data
{ "name": "Joe-Nelson" }
Query
create first_name from name.arraySplit("-")[0]
| create last_name from name.arraySplit("-")[1]
Result
{ first_name: Joe, last_name: Nelson }
arrayJoin – Join array values into a string
Description
Use arrayJoin to convert an array into a readable string using a delimiter.
Syntax
<array_field>.arrayJoin(<delimiter>)
Example – Format a user action log
Input data
{ "users": ["Emma", "Ofri", "Zev"], "action": "LOGIN" }
Query
create msg from `Users {users.arrayJoin(', ')} performed action {action}`
Result
{ msg: Users Emma, Ofri, Zev performed action LOGIN }
urlDecode / urlEncode – Decode or encode URL-safe strings
Description
Use urlDecode to make encoded strings readable. Use urlEncode when you need to safely transmit or store text.
Syntax
<field>.urlDecode()
<field>.urlEncode()
Example – Decode a query string parameter
Input data
{ "query_string_parameters": { "name": "Tom%20Kosman" } }
Query
replace query_string_parameters.name with query_string_parameters.name.urlDecode()
Result
{ query_string_parameters: { name: Tom Kosman } }
decodeBase64 – Decode base64 strings
Description
Use decodeBase64 to convert encoded strings (e.g., compressed URLs or payloads) into readable values.
Syntax
<field>.decodeBase64()
Example – Decode a base64 URL
Input data
{ "full_url_encoded": "aHR0cHM6Ly9jb3JhbG9naXguY29t" }
Query
create full_url from full_url_encoded.decodeBase64()
Result
{ full_url: https://coralogix.com }
contains, startsWith – Check for string patterns
Description
Use contains or startsWith to detect substrings in a string field. Useful for filtering by prefix, domain, or label.
Syntax
<field>.contains(<substring>)
<field>.startsWith(<substring>)
Example – Identify log types or domains
Input data
{ "url": "https://google.com/search", "msg": "ERROR timeout exceeded" }
Query
filter url.contains("google.com")
| create is_error_log from msg.startsWith("ERROR")
Result
{ is_error_log: true }
choose – Flatten deeply nested fields
Description
Use choose to extract a deeply nested field and bring it to the top level. Makes it easier to read and query.
Syntax
choose <deep_field> as <alias>
Example – Simplify metric access
Input data
{
"http_request": {
"metrics": {
"bytes_metrics": {
"bytes_received": 1024
}
}
}
}
Query
choose http_request.metrics.bytes_metrics.bytes_received as bytes_received
Result
{ bytes_received: 1024 }
extract ... using kv() – Parse key-value strings
Description
Use extract ... using kv() to convert a structured string into a map of fields. Add datatypes to cast values into numbers or timestamps.
There are several extractor functions that can be used with the extract command.
Syntax
extract <field> into <object> using kv(pair_delimiter, key_delimiter) datatypes ...
Example – Parse and cast query parameters
Input data
{ "msg": "query_id=200&duration_ms=1000" }
Query
extract msg into data using kv(pair_delimiter='&', key_delimiter='=') datatypes duration_ms:number
Result
{ data: { query_id: 200, duration_ms: 1000 } }
Bracket notation – Access special keypaths
Description
Use bracket notation to access keys that contain dots, spaces, or special characters. Required in archive/compliance mode.
Syntax
<root_field>['key.with.dot.or.special-character']
Example – Filter by a key with dots
Input data
{ "k8s.tags.process_id": 1234 }
Query
filter k8s['tags.process_id'] == 1234
Result
{ k8s.tags.process_id: 1234 }
Common pitfalls or gotchas
- Exploding arrays removes other fields unless you preserve them. Always use
original preserveunless you want a minimal result. - String length limits can break string functions. If a field is longer than 256 characters, some functions may silently return
nullin high-tier mode. - Always quote bracketed keypaths. Even one missed bracket or dot can break your query.