setDiff

Description

Returns the set difference of two arrays, producing a new array with elements from array1 that are not in array2.

• Duplicates are discarded when computing the difference.
• Supported element types include string, bool, number, interval, timestamp, regexp, and enum.

Syntax

Like many functions in DataPrime, setDiff supports two notations, function and method notation. These interchangeable forms allow flexibility in how you structure expressions.

setDiff(array1: array<T>, array2: array<T>): array<T>

(array1: array<T>).setDiff(array2: array<T>): array<T>

Arguments

Example

Use case: Identify unauthorized IP addresses

Suppose you have aggregated a list of IP addresses that accessed different paths. Consider the following input:

{
    "path": "/home",
    "ip_addresses": ["156.76.87.4", "156.76.12.4", "156.74.1.4"]
},
{
    "path": "/checkout",
    "ip_addresses": ["156.76.87.4"]
}

By applying setDiff, you can compare the observed IP addresses against a known allow list to identify unauthorized addresses.

Example query

create unauthorized_ip_addresses from setDiff(ip_addresses, ["156.76.12.4", "156.76.87.4"])

create unauthorized_ip_addresses from ip_addresses.setDiff(["156.76.12.4", "156.76.87.4"])

Example output

The result will include a new field unauthorized_ip_addresses containing only the addresses not in the allow list:

{
    "path": "/home",
    "ip_addresses": ["156.76.87.4", "156.76.12.4", "156.74.1.4"],
    "unauthorized_ip_addresses": ["156.74.1.4"]
},
{
    "path": "/checkout",
    "ip_addresses": ["156.76.87.4"],
    "unauthorized_ip_addresses": []
}