## Overview HTTP API queries are made using JSON over HTTPS (referred to here as HTTP requests). These API calls can be made with a variety of tools and programming languages. Using the API provides programmatic access to Coralogix's data archive, allowing users to automate queries, integrate with other systems, and perform large-scale or time-sensitive operations that would be less efficient or impractical through the UI. ## Requirements All requests must use HTTPS. Calls made over plain HTTP will fail. To use this API you need to create a [personal or team API key](https://coralogix.com/docs/user-guides/account-management/api-keys/api-keys/index.md). It’s recommended to use permission presets, as they are automatically updated with all relevant permissions. Alternatively, you can manually add individual permissions. Any request that lacks proper authentication will be rejected. To authenticate, add an `Authorization Bearer ` header to your API request. It contains a Bearer Token, which identifies a single user, bot user, or workspace-application relationship for authentication. Select the https://api.\[[DOMAIN_VALUE]\]/api/v1/dataprime/query endpoint that corresponds to your Coralogix [domain](https://coralogix.com/docs/user-guides/account-management/account-settings/coralogix-domain/index.md) using the domain selector at the top of the page. ## Request payload body When submitting data to a resource via `POST`, you must submit your payload in JSON. The only required field in the API body is `query`. All other fields are optional and form the metadata object. Note Background queries use a different request structure (no `metadata` wrapper) and have their own required fields. See [Background queries HTTP API overview](#background-queries-http-api-overview) below. The following example consists of a JSON object that represents the request: ```json { "query": "source logs | limit 100", "metadata": { // metadata object is optional "tier": "TIER_ARCHIVE", // Search target of the query. Options: TIER_ARCHIVE, TIER_FREQUENT_SEARCH. Default: TIER_FREQUENT_SEARCH "syntax": "QUERY_SYNTAX_DATAPRIME", // Query syntax. Options: QUERY_SYNTAX_DATAPRIME, QUERY_SYNTAX_LUCENE, others (see Swagger). Default: QUERY_SYNTAX_DATAPRIME "startDate": "2023-05-29T11:20:00.00Z", // ISO 8601 date-time string defining the beginning of the query timeframe "endDate": "2023-05-29T11:30:00.00Z", // ISO 8601 date-time string defining the ending of the query timeframe "defaultSource": "logs", // Default source used when source is omitted in a query "limit": 100, // int32; limits the number of returned results. Max: 50,000 (TIER_ARCHIVE) or 12,000 (TIER_FREQUENT_SEARCH). Default: 2000 "strictFieldsValidation": false, // true = fail on unknown fields. Default: false "nowDate": "2023-05-29T12:00:00.00Z" // substitutes for now() in query. Default: current time }, "executionProfile": { // optional; controls the query engine's accuracy/performance trade-off "preset": "EXECUTION_PROFILE_PRESET_ACCURACY" // EXECUTION_PROFILE_PRESET_ACCURACY (default): fresh results from source. EXECUTION_PROFILE_PRESET_PERFORMANCE: use caches for faster, approximate results } } ``` For the full request and response schema reference (including all metadata fields, enum values, and limits), see the [DataPrime Query Service Swagger reference](https://coralogix.com/docs/static/swagger/index.md). Note `startDate` and `endDate` must be valid ISO 8601 date-time strings. If your local time zone is not UTC, either convert the timestamp to UTC (e.g., `2023-05-29T18:20:00.00Z`) or include the time zone offset (e.g., `2023-05-29T11:20:00.00-07:00` for Pacific Daylight Time, `2023-05-29T11:20:00.00+05:30` for India Standard Time). Note When querying **archived** logs using the API, you must explicitly specify `"tier": "TIER_ARCHIVE"` in the metadata object of your request. Otherwise, the default is `"tier": "TIER_FREQUENT_SEARCH"`. ## Direct API query **Endpoint:** `https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/query` ```json { "query": "source logs | limit 10" } ``` ### Make the API call ```bash curl --location 'https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/query' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{ "query": "source logs | limit 10" }' ``` Note When working with `curl` or similar command-line tools to make JSON over HTTP API requests, it's important to understand how your shell (typically Bash) processes quotes inside command arguments. This becomes especially relevant when your request payload includes JSON — and even more so when the JSON contains both single and double quotes. Find out more [here](https://coralogix.com/docs/dataprime/API/quoting-json/index.md). ```python import requests import json endpoint = "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/query" api_key = "" headers = { "Content-Type": "application/json", "Authorization": f"Bearer {api_key}" } payload = {"query": "source logs | limit 10"} response = requests.post(endpoint, headers=headers, json=payload) if response.status_code == 200: try: for part in response.text.strip().split("\n"): print(json.loads(part)) except json.JSONDecodeError: print("Response is not valid JSON:") print(response.text) else: print(f"Request failed with status code {response.status_code}") print(response.text) ``` ```javascript const endpoint = 'https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/query'; const apiKey = ''; const payload = { query: 'source logs | limit 10', }; fetch(endpoint, { method: 'POST', headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${apiKey}`, }, body: JSON.stringify(payload), }) .then(async (res) => { const text = await res.text(); if (!res.ok) throw new Error(`HTTP ${res.status}`); console.log('Status:', res.status); console.log('Response:', text); }) .catch(err => { console.error('Error:', err); }); ``` ```go package main import ( "bytes" "fmt" "io" "net/http" ) func main() { url := "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/query" apiKey := "" payload := []byte(`{ "query": "source logs | limit 10" }`) req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload)) if err != nil { fmt.Println("Error creating request:", err) return } req.Header.Set("Content-Type", "application/json") req.Header.Set("Authorization", "Bearer "+apiKey) client := &http.Client{} resp, err := client.Do(req) if err != nil { fmt.Println("Request error:", err) return } defer resp.Body.Close() body, err := io.ReadAll(resp.Body) if err != nil { fmt.Println("Error reading response:", err) return } fmt.Println("Status:", resp.Status) fmt.Println("Response:") fmt.Println(string(body)) } ``` ```php "; $payload = [ "query" => "source logs | limit 10" ]; $ch = curl_init($url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_HTTPHEADER, [ 'Content-Type: application/json', "Authorization: Bearer $apiKey" ]); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($payload)); $response = curl_exec($ch); $httpStatus = curl_getinfo($ch, CURLINFO_HTTP_CODE); curl_close($ch); echo "Status: $httpStatus\n"; echo "Response: $response\n"; ?> ``` ```java import java.io.*; import java.net.HttpURLConnection; import java.net.URL; public class CoralogixQuery { public static void main(String[] args) throws Exception { String endpoint = "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/query"; String apiKey = ""; URL url = new URL(endpoint); HttpURLConnection con = (HttpURLConnection) url.openConnection(); con.setRequestMethod("POST"); con.setRequestProperty("Content-Type", "application/json"); con.setRequestProperty("Authorization", "Bearer " + apiKey); con.setDoOutput(true); String payload = "{\"query\": \"source logs | limit 10\"}"; try (OutputStream os = con.getOutputStream()) { byte[] input = payload.getBytes("utf-8"); os.write(input, 0, input.length); } int status = con.getResponseCode(); InputStream responseStream = (status < 400) ? con.getInputStream() : con.getErrorStream(); BufferedReader in = new BufferedReader(new InputStreamReader(responseStream)); String line; StringBuilder response = new StringBuilder(); while ((line = in.readLine()) != null) { response.append(line.trim()); } in.close(); System.out.println("Status: " + status); System.out.println("Response: " + response.toString()); } } ``` ```csharp using System; using System.Net.Http; using System.Text; using System.Threading.Tasks; class Program { static async Task Main() { var client = new HttpClient(); var url = "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/query"; var apiKey = ""; var payload = "{\"query\": \"source logs | limit 10\"}"; var content = new StringContent(payload, Encoding.UTF8, "application/json"); client.DefaultRequestHeaders.Add("Authorization", $"Bearer {apiKey}"); var response = await client.PostAsync(url, content); var result = await response.Content.ReadAsStringAsync(); Console.WriteLine($"Status: {(int)response.StatusCode}"); Console.WriteLine("Response:"); Console.WriteLine(result); } } ``` ### Usage examples **Limit the number of results** Use `limit` to cap the number of rows returned, for example when sampling data or testing a query before running it at full scale: ```bash curl --location 'https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/query' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{ "query": "source logs", "metadata": { "tier": "TIER_ARCHIVE", "syntax": "QUERY_SYNTAX_DATAPRIME", "startDate": "2023-05-29T11:00:00.00Z", "endDate": "2023-05-29T11:30:00.00Z", "limit": 500 } }' ``` **Enforce strict field validation** Set `strictFieldsValidation` to `true` to immediately fail when a query references a field not present in your schema. This is useful in automated scripts or CI pipelines where silent schema mismatches should surface as errors rather than warnings: ```bash curl --location 'https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/query' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{ "query": "source logs | filter nonexistent_field == '\''value'\''", "metadata": { "tier": "TIER_ARCHIVE", "syntax": "QUERY_SYNTAX_DATAPRIME", "startDate": "2023-05-29T11:00:00.00Z", "endDate": "2023-05-29T11:30:00.00Z", "strictFieldsValidation": true } }' ``` On failure, the response is a compilation error that pinpoints the unknown field by its position in the query: ```text Compilation errors: - keypath does not exist at [0:21-0:38]: nonexistent_field ``` **Pin the reference time for `now()`** Use `nowDate` when your query uses `now()` and you need a reproducible, fixed reference time — for example, when replaying historical queries or running scheduled reports: ```bash curl --location 'https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/query' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{ "query": "source logs | filter $m.timestamp > now() - 1h", "metadata": { "tier": "TIER_ARCHIVE", "syntax": "QUERY_SYNTAX_DATAPRIME", "startDate": "2023-05-29T10:00:00.00Z", "endDate": "2023-05-29T12:00:00.00Z", "nowDate": "2023-05-29T12:00:00.00Z" } }' ``` ### API response **Status:** `200 OK` **Content-Type:** `application/x-ndjson` (newline-delimited JSON) **Body format:** The response is NDJSON; each line is a JSON object: 1. `{"queryId": {...}}` — the query identifier 1. `{"result": {"results": [...]}}` — the DataPrime query results (returned in batches; each batch can include multiple rows) 1. `{"statistics": {...}}` — query execution statistics, returned at the end of the response (see [Query statistics](#query-statistics)) ```http HTTP/1.1 200 OK Content-Type: application/x-ndjson {"queryId":{"queryId":"12345678-07f8-4313-1234-d954b1b45d31"}} {"result":{"results":[{"metadata":[{"key":"logid","value":"c3ca5343-88dc-4807-a8f3-82832274afb7"}],"labels":[{"key":"applicationname","value":"staging"}],"userData":"{ ... \"log_obj\":{\"level\":\"INFO\",\"message\":\"some log message\" ... }, ...}"}]}} {"statistics":{"status":"COMPLETED","e2eDurationMs":"260","outputRowCount":"1","storage":{"objectStore":{"bytesRead":"0","crossRegionBytesRead":"0","limits":{"scan":{"reached":false,"limit":"50000000000","measuredValue":"454108"},"shuffleSize":{"reached":false,"limit":"1000000000","measuredValue":"704"},"filesRead":{"reached":false,"measuredValue":"0"},"column":{"reached":false}}}}}} ``` For the full response schema (including all `result` and `error` field shapes), see the [`QueryResponse` reference](https://coralogix.com/docs/static/swagger/#schemacom.coralogixapis.dataprime.v1.queryresponse). #### Query statistics Every direct query response ends with a `statistics` object that describes the query's execution outcome and resource usage. It is the same statistics model recorded in the [`engine.queries`](https://coralogix.com/docs/user-guides/data-layer/system_dataspace/engine_queries/index.md) dataset — the API returns it live, so you don't have to query that dataset separately. Numeric fields are JSON strings Integer values (durations, byte counts, row counts, limit thresholds) appear as **JSON strings** in this response — for example, `"260"` and `"50000000000"`. This preserves 64-bit precision. Parse them as integers in your client. | Field | Description | | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `status` | Final status: `COMPLETED`, `FAILED`, `CANCELLED`, `TIMED_OUT`, or `INCOMPLETE`. | | `e2eDurationMs` | End-to-end execution time in milliseconds. | | `outputRowCount` | Number of rows returned by the query. | | `storage.objectStore.bytesRead` | Bytes read from customer-exposed storage (excludes cache and staging reads). | | `storage.objectStore.crossRegionBytesRead` | Bytes read from a region different from the query engine's region. A nonzero value indicates inter-region transfer cost. | | `storage.objectStore.limits.` | Per-resource limit signals. Resources include `scan`, `shuffleSize`, `filesRead`, and `column`. Each entry includes `reached` (`true` when the limit applies) and, where applicable, `limit` (the threshold) and `measuredValue` (the observed amount). Threshold values vary by tier and account configuration. | Programmatic limit handling Inspect `storage.objectStore.limits.*.reached` after each call and, when a limit applies, automatically reduce the time range or scope and reissue the query — no human intervention needed. The same `statistics` model is also returned by [`GetBackgroundQueryStatus`](#get-background-query-status) once a background query reaches a terminated state. ## Background queries HTTP API overview The Background Queries JSON over HTTP API enables you to run high-latency, long-running queries asynchronously using DataPrime or Lucene syntax from scripts or the CLI. Designed for recurring, extensive analytical tasks—such as monthly or quarterly reports—this API allows you to offload complex queries to run in the background, freeing you to continue real-time monitoring within the Coralogix platform. Use it to: - Seamlessly query archived logs and spans, regardless of time range. - Leverage higher scan, file, and shuffle limits for deeper data access. - Take advantage of extended execution times for heavy analytical workloads. The API supports the following background gRPCs: - [Submit a background query](#submit-a-background-query) - [Get background query status](#get-background-query-status) - [Cancel a background query](#cancel-a-background-query) - [Get background query data](#get-background-query-data) ## Submit a background query **Endpoint:** `https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query` In addition to the `query` field, `syntax`, `startDate`, and `endDate` are required fields for background queries. Note The structure of background query requests differs from that of direct query requests: background query requests don't contain the `metadata` nested object. The optional `defaultSource`, `name`, `description`, and `resultsDestination` fields are top-level — see [Writing results to a dataset](#writing-results-to-a-dataset). ```json { "query": "source logs", "syntax": "QUERY_SYNTAX_DATAPRIME", // QUERY_SYNTAX_LUCENE for Lucene "startDate": "2025-01-20T00:00:00Z", // UTC "endDate": "2025-01-20T01:00:00Z", // UTC "nowDate": "2025-01-20T02:00:00Z", // (optional field) UTC } ``` The following optional top-level fields are also supported: | **Field** | **Description** | **Example** | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | | `resultsDestination` | Where to write the query results. See [Writing results to a dataset](#writing-results-to-a-dataset). When omitted, results go to a temporary dataset retained for 30 days. | | | `defaultSource` | Fully qualified name of the default source to use when the query omits one. Defaults to `default/logs`. | `default/logs` | | `name` | Name for the query. Must start with up to two underscores followed by a letter, then contain only letters, numbers, underscores, or dots; no consecutive or trailing dots; between 2 and 255 characters. Defaults to an auto-generated name. | `monthly_report` | | `description` | Free-text description, up to 2,048 characters. | `Monthly archive rollup` | ### Make the API call ```bash curl --location 'https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{ "query": "source logs | limit 101", "syntax": "QUERY_SYNTAX_DATAPRIME", "startDate": "2025-01-20T00:00:00Z", "endDate": "2025-01-20T01:00:00Z", "nowDate": "2025-01-20T02:00:00Z" }' ``` ```python import requests import json endpoint = "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query" api_key = "" headers = { "Content-Type": "application/json", "Authorization": f"Bearer {api_key}" } payload = {"query": "source logs | limit 10"} response = requests.post(endpoint, headers=headers, json=payload) if response.status_code == 200: try: for part in response.text.strip().split("\n"): print(json.loads(part)) except json.JSONDecodeError: print("Response is not valid JSON:") print(response.text) else: print(f"Request failed with status code {response.status_code}") print(response.text) ``` ```javascript const endpoint = 'https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query'; const apiKey = ''; const payload = { query: "source logs | limit 101", syntax: "QUERY_SYNTAX_DATAPRIME", startDate: "2025-01-20T00:00:00Z", endDate: "2025-01-20T01:00:00Z", nowDate: "2025-01-20T02:00:00Z" }; fetch(endpoint, { method: 'POST', headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${apiKey}`, }, body: JSON.stringify(payload), }) .then(async (res) => { const text = await res.text(); if (!res.ok) throw new Error(`HTTP ${res.status}`); console.log('Status:', res.status); console.log('Response:', text); }) .catch(err => { console.error('Error:', err); }); ``` ```go package main import ( "bytes" "fmt" "io" "net/http" ) func main() { url := "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query" apiKey := "" payload := []byte(`{ "query": "source logs | limit 101", "syntax": "QUERY_SYNTAX_DATAPRIME", "startDate": "2025-01-20T00:00:00Z", "endDate": "2025-01-20T01:00:00Z", "nowDate": "2025-01-20T02:00:00Z" }`) req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload)) if err != nil { fmt.Println("Error creating request:", err) return } req.Header.Set("Content-Type", "application/json") req.Header.Set("Authorization", "Bearer "+apiKey) client := &http.Client{} resp, err := client.Do(req) if err != nil { fmt.Println("Request error:", err) return } defer resp.Body.Close() body, err := io.ReadAll(resp.Body) if err != nil { fmt.Println("Error reading response:", err) return } fmt.Println("Status:", resp.Status) fmt.Println("Response:") fmt.Println(string(body)) } ``` ```php "; $payload = [ "query" => "source logs | limit 101", "syntax" => "QUERY_SYNTAX_DATAPRIME", "startDate" => "2025-01-20T00:00:00Z", "endDate" => "2025-01-20T01:00:00Z", "nowDate" => "2025-01-20T02:00:00Z" ]; $ch = curl_init($url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_HTTPHEADER, [ 'Content-Type: application/json', "Authorization: Bearer $apiKey" ]); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($payload)); $response = curl_exec($ch); $httpStatus = curl_getinfo($ch, CURLINFO_HTTP_CODE); curl_close($ch); echo "Status: $httpStatus\n"; echo "Response: $response\n"; ?> ``` ```java import java.io.*; import java.net.HttpURLConnection; import java.net.URL; public class CoralogixQuery { public static void main(String[] args) throws Exception { String endpoint = "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query"; String apiKey = ""; URL url = new URL(endpoint); HttpURLConnection con = (HttpURLConnection) url.openConnection(); con.setRequestMethod("POST"); con.setRequestProperty("Content-Type", "application/json"); con.setRequestProperty("Authorization", "Bearer " + apiKey); con.setDoOutput(true); String payload = """ { "query": "limit 101", "syntax": "QUERY_SYNTAX_DATAPRIME", "startDate": "2025-01-20T00:00:00Z", "endDate": "2025-01-20T01:00:00Z", "nowDate": "2025-01-20T02:00:00Z" } """; try (OutputStream os = con.getOutputStream()) { byte[] input = payload.getBytes("utf-8"); os.write(input, 0, input.length); } int status = con.getResponseCode(); InputStream responseStream = (status < 400) ? con.getInputStream() : con.getErrorStream(); BufferedReader in = new BufferedReader(new InputStreamReader(responseStream)); String line; StringBuilder response = new StringBuilder(); while ((line = in.readLine()) != null) { response.append(line.trim()); } in.close(); System.out.println("Status: " + status); System.out.println("Response: " + response.toString()); } } ``` ```csharp using System; using System.Net.Http; using System.Text; using System.Threading.Tasks; class Program { static async Task Main() { var client = new HttpClient(); var url = "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query"; var apiKey = ""; var payload = "{" + "\"query\": \"limit 101\"," + "\"syntax\": \"QUERY_SYNTAX_DATAPRIME\"," + "\"startDate\": \"2025-01-20T00:00:00Z\"," + "\"endDate\": \"2025-01-20T01:00:00Z\"," + "\"nowDate\": \"2025-01-20T02:00:00Z\"" + "}"; var content = new StringContent(payload, Encoding.UTF8, "application/json"); client.DefaultRequestHeaders.Add("Authorization", $"Bearer {apiKey}"); var response = await client.PostAsync(url, content); var result = await response.Content.ReadAsStringAsync(); Console.WriteLine($"Status: {(int)response.StatusCode}"); Console.WriteLine("Response:"); Console.WriteLine(result); } } ``` ### Response **Status:** `200 OK` **Content-Type:** `application/json` **Body format:** A single JSON object containing the background query identifier and any non-fatal warnings. - `queryId` — unique ID used for follow-up operations (e.g., check status, fetch results, cancel). - `warnings` — optional array of warning messages. ```http HTTP/1.1 200 OK Content-Type: application/json { "queryId": "a63653bc-aba8-49cf-b4a4-1b1ca9907f54", "warnings": [] } ``` > Notes > > - This response does **not** include results; use the `queryId` for subsequent calls to retrieve status or results. > - If `warnings` is present and non-empty, the query was accepted but with caveats (e.g., deprecated fields, partial filters). Response status code: 200 ## Writing results to a dataset By default, a background query writes its results to a temporary, engine-owned dataset (`system/engine.resultsets.bg_queries.{unique_id}`) that is retained for 30 days. To keep results longer or reuse them across the platform, write them to a persistent, user-defined dataset with the optional `resultsDestination` object. The target dataset must already exist under the `default` dataspace — create it first in [Dataspace Management](https://coralogix.com/docs/user-guides/data-layer/default-dataspace/user-defined-datasets/#create-a-dataset). For the in-app equivalent, see [Background Queries](https://coralogix.com/docs/user-guides/dataengine/background_queries_v2/index.md). `resultsDestination` accepts **at most one** of `temporaryDestination` or `persistentDestination`: ```json { "query": "source logs | limit 1000", "syntax": "QUERY_SYNTAX_DATAPRIME", "startDate": "2025-01-20T00:00:00Z", "endDate": "2025-01-20T01:00:00Z", "resultsDestination": { "persistentDestination": { "datasets": [ { "dataspace": "default", "dataset": "my_summary_dataset", "writeMode": "DATASET_WRITE_MODE_APPEND" } ], "partitioning": { "dateHourPartitioned": { "dataprimePath": "$m.timestamp" } } } } } ``` | **Field** | **Description** | **Example** | | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- | | `temporaryDestination` | Store results in the engine-owned `system/engine.resultsets.bg_queries.{unique_id}` dataset for 30 days. Pass an empty object. This is the default when `resultsDestination` is omitted. | `{}` | | `persistentDestination.datasets[].dataspace` | Dataspace of the target dataset. Must be `default` — background queries write only to the default dataspace. | `default` | | `persistentDestination.datasets[].dataset` | Name of the target dataset. The dataset must already exist; this API does not create datasets. Create it first in [Dataspace Management](https://coralogix.com/docs/user-guides/data-layer/default-dataspace/user-defined-datasets/#create-a-dataset). | `my_summary_dataset` | | `persistentDestination.datasets[].writeMode` | `DATASET_WRITE_MODE_APPEND` adds rows to the current dataset version; `DATASET_WRITE_MODE_OVERWRITE` starts a new version while retaining earlier ones. | `DATASET_WRITE_MODE_APPEND` | | `persistentDestination.partitioning` | Optional. Accepts at most one of `unpartitioned` (an empty object) or `dateHourPartitioned`. When omitted, results use the same partitioning as the query's results. | | | `persistentDestination.partitioning.dateHourPartitioned.dataprimePath` | DataPrime keypath of a timestamp field to partition on. Must exist in the query's results and be of timestamp type. | `$m.timestamp` | Note `persistentDestination.datasets` accepts at most one entry. Submitting more than one returns a `BAD_REQUEST` error. Note Appending the same raw data more than once creates duplicate records — there is no automatic deduplication by log ID. ## Get background query status **Endpoint:** `https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/status` ```json { "queryId": "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64" } ``` ### Make the API call ```bash curl --location 'https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/status' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{"queryId": "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64"}' ``` ```python import requests import json endpoint = "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/status" api_key = "" headers = { "Content-Type": "application/json", "Authorization": f"Bearer {api_key}" } payload = {"queryId": "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64"} response = requests.post(endpoint, headers=headers, json=payload) if response.status_code == 200: try: for part in response.text.strip().split("\n"): print(json.loads(part)) except json.JSONDecodeError: print("Response is not valid JSON:") print(response.text) else: print(f"Request failed with status code {response.status_code}") print(response.text) ``` ```javascript const endpoint = 'https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/status'; const apiKey = ''; const payload = {queryId: "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64"}; fetch(endpoint, { method: 'POST', headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${apiKey}`, }, body: JSON.stringify(payload), }) .then(async (res) => { const text = await res.text(); if (!res.ok) throw new Error(`HTTP ${res.status}`); console.log('Status:', res.status); console.log('Response:', text); }) .catch(err => { console.error('Error:', err); }); ``` ```go package main import ( "bytes" "fmt" "io" "net/http" ) func main() { url := "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/status" apiKey := "" payload := []byte(`{"queryId": "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64"}`) req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload)) if err != nil { fmt.Println("Error creating request:", err) return } req.Header.Set("Content-Type", "application/json") req.Header.Set("Authorization", "Bearer "+apiKey) client := &http.Client{} resp, err := client.Do(req) if err != nil { fmt.Println("Request error:", err) return } defer resp.Body.Close() body, err := io.ReadAll(resp.Body) if err != nil { fmt.Println("Error reading response:", err) return } fmt.Println("Status:", resp.Status) fmt.Println("Response:") fmt.Println(string(body)) } ``` ```php "; $payload = ["queryId" => "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64"]; $ch = curl_init($url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_HTTPHEADER, [ 'Content-Type: application/json', "Authorization: Bearer $apiKey" ]); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($payload)); $response = curl_exec($ch); $httpStatus = curl_getinfo($ch, CURLINFO_HTTP_CODE); curl_close($ch); echo "Status: $httpStatus\n"; echo "Response: $response\n"; ?> ``` ```java import java.io.*; import java.net.HttpURLConnection; import java.net.URL; public class CoralogixQuery { public static void main(String[] args) throws Exception { String endpoint = "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/status"; String apiKey = ""; URL url = new URL(endpoint); HttpURLConnection con = (HttpURLConnection) url.openConnection(); con.setRequestMethod("POST"); con.setRequestProperty("Content-Type", "application/json"); con.setRequestProperty("Authorization", "Bearer " + apiKey); con.setDoOutput(true); String payload = "{\"queryId\": \"c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64\"}"; try (OutputStream os = con.getOutputStream()) { byte[] input = payload.getBytes("utf-8"); os.write(input, 0, input.length); } int status = con.getResponseCode(); InputStream responseStream = (status < 400) ? con.getInputStream() : con.getErrorStream(); BufferedReader in = new BufferedReader(new InputStreamReader(responseStream)); String line; StringBuilder response = new StringBuilder(); while ((line = in.readLine()) != null) { response.append(line.trim()); } in.close(); System.out.println("Status: " + status); System.out.println("Response: " + response.toString()); } } ``` ```csharp using System; using System.Net.Http; using System.Text; using System.Threading.Tasks; class Program { static async Task Main() { var client = new HttpClient(); var url = "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/status"; var apiKey = ""; var payload = "{\"queryId\": \"c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64\"}" var content = new StringContent(payload, Encoding.UTF8, "application/json"); client.DefaultRequestHeaders.Add("Authorization", $"Bearer {apiKey}"); var response = await client.PostAsync(url, content); var result = await response.Content.ReadAsStringAsync(); Console.WriteLine($"Status: {(int)response.StatusCode}"); Console.WriteLine("Response:"); Console.WriteLine(result); } } ``` ### Response **Status:** `200 OK` **Content-Type:** `application/json` **Body format:** A single JSON object describing the background query’s lifecycle and diagnostics. - `terminated` — present when the query has finished. - `runningSince` — ISO-8601 UTC timestamp when execution started. - `terminatedAt` — ISO-8601 UTC timestamp when execution ended. - `success` — empty object `{}` indicating successful completion. - `submittedAt` — ISO-8601 UTC timestamp when the job was queued. - `metadata` — array of diagnostic entries. - `statistics.bytesScanned` — total bytes scanned (returned as a string). - `warnings` — optional array of non-fatal warning messages. ```http HTTP/1.1 200 OK Content-Type: application/json { "terminated": { "runningSince": "2025-01-23T15:16:01Z", "terminatedAt": "2025-01-23T15:16:01Z", "success": {} }, "submittedAt": "2025-01-23T15:15:58Z", "metadata": [ { "statistics": { "bytesScanned": "506070" } } ], "warnings": [] } ``` > Notes > > - If the query is **still running**, `terminated` may be absent; check for that to determine state. > - Treat `success` as a boolean flag (present → success). If your client expects failures, also check for alternative fields your API may return (e.g., `error`). > - `bytesScanned` is a string to avoid integer size issues; convert to a number if needed. ## Cancel a background query **Endpoint:** `https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/cancel` ```json { "queryId": "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64" } ``` ### Make the API call ```bash curl --location 'https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/cancel' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{"queryId": "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64"}' ``` ```python import requests import json endpoint = "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/cancel" api_key = "" headers = { "Content-Type": "application/json", "Authorization": f"Bearer {api_key}" } payload = {"queryId": "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64"} response = requests.post(endpoint, headers=headers, json=payload) if response.status_code == 200: try: for part in response.text.strip().split("\n"): print(json.loads(part)) except json.JSONDecodeError: print("Response is not valid JSON:") print(response.text) else: print(f"Request failed with status code {response.status_code}") print(response.text) ``` ```javascript const endpoint = 'https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/cancel'; const apiKey = ''; const payload = {queryId: "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64"}; fetch(endpoint, { method: 'POST', headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${apiKey}`, }, body: JSON.stringify(payload), }) .then(async (res) => { const text = await res.text(); if (!res.ok) throw new Error(`HTTP ${res.status}`); console.log('Status:', res.status); console.log('Response:', text); }) .catch(err => { console.error('Error:', err); }); ``` ```go package main import ( "bytes" "fmt" "io" "net/http" ) func main() { url := "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/cancel" apiKey := "" payload := []byte(`{"queryId": "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64"}`) req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload)) if err != nil { fmt.Println("Error creating request:", err) return } req.Header.Set("Content-Type", "application/json") req.Header.Set("Authorization", "Bearer "+apiKey) client := &http.Client{} resp, err := client.Do(req) if err != nil { fmt.Println("Request error:", err) return } defer resp.Body.Close() body, err := io.ReadAll(resp.Body) if err != nil { fmt.Println("Error reading response:", err) return } fmt.Println("Status:", resp.Status) fmt.Println("Response:") fmt.Println(string(body)) } ``` ```php "; $payload = ["queryId" => "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64"]; $ch = curl_init($url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_HTTPHEADER, [ 'Content-Type: application/json', "Authorization: Bearer $apiKey" ]); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($payload)); $response = curl_exec($ch); $httpStatus = curl_getinfo($ch, CURLINFO_HTTP_CODE); curl_close($ch); echo "Status: $httpStatus\n"; echo "Response: $response\n"; ?> ``` ```java import java.io.*; import java.net.HttpURLConnection; import java.net.URL; public class CoralogixQuery { public static void main(String[] args) throws Exception { String endpoint = "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/cancel"; String apiKey = ""; URL url = new URL(endpoint); HttpURLConnection con = (HttpURLConnection) url.openConnection(); con.setRequestMethod("POST"); con.setRequestProperty("Content-Type", "application/json"); con.setRequestProperty("Authorization", "Bearer " + apiKey); con.setDoOutput(true); String payload = "{\"queryId\": \"c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64\"}"; try (OutputStream os = con.getOutputStream()) { byte[] input = payload.getBytes("utf-8"); os.write(input, 0, input.length); } int status = con.getResponseCode(); InputStream responseStream = (status < 400) ? con.getInputStream() : con.getErrorStream(); BufferedReader in = new BufferedReader(new InputStreamReader(responseStream)); String line; StringBuilder response = new StringBuilder(); while ((line = in.readLine()) != null) { response.append(line.trim()); } in.close(); System.out.println("Status: " + status); System.out.println("Response: " + response.toString()); } } ``` ```csharp using System; using System.Net.Http; using System.Text; using System.Threading.Tasks; class Program { static async Task Main() { var client = new HttpClient(); var url = "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/cancel"; var apiKey = ""; var payload = "{\"queryId\": \"c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64\"}" var content = new StringContent(payload, Encoding.UTF8, "application/json"); client.DefaultRequestHeaders.Add("Authorization", $"Bearer {apiKey}"); var response = await client.PostAsync(url, content); var result = await response.Content.ReadAsStringAsync(); Console.WriteLine($"Status: {(int)response.StatusCode}"); Console.WriteLine("Response:"); Console.WriteLine(result); } } ``` ### Response **Status:** `200 OK` **Content-Type:** `application/json` **Body format:** On success, the endpoint returns an **empty JSON object**. ```json {} ``` > Notes > > - Cancellation is typically **idempotent**: canceling an already finished or already canceled query will also return `200 {}`. > - If cancellation is asynchronous, use the **Get Background Query Status** endpoint with the same `queryId` to confirm that the job has transitioned to a terminated state. > - Some implementations may return other codes (e.g., `404` for unknown `queryId`, `409` if the job cannot be canceled). ## Get background query data **Endpoint:** `https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/data` ```json { "queryId": "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64" } ``` Status code: 200 ### Make the API call ```bash curl --location 'https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/data' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{"queryId": "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64"}' ``` ```python import requests import json endpoint = "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/data" api_key = "" headers = { "Content-Type": "application/json", "Authorization": f"Bearer {api_key}" } payload = {"queryId": "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64"} response = requests.post(endpoint, headers=headers, json=payload) if response.status_code == 200: try: for part in response.text.strip().split("\n"): print(json.loads(part)) except json.JSONDecodeError: print("Response is not valid JSON:") print(response.text) else: print(f"Request failed with status code {response.status_code}") print(response.text) ``` ```javascript const endpoint = 'https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/data'; const apiKey = ''; const payload = {queryId: "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64"}; fetch(endpoint, { method: 'POST', headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${apiKey}`, }, body: JSON.stringify(payload), }) .then(async (res) => { const text = await res.text(); if (!res.ok) throw new Error(`HTTP ${res.status}`); console.log('Status:', res.status); console.log('Response:', text); }) .catch(err => { console.error('Error:', err); }); ``` ```go package main import ( "bytes" "fmt" "io" "net/http" ) func main() { url := "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/data" apiKey := "" payload := []byte(`{"queryId": "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64"}`) req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload)) if err != nil { fmt.Println("Error creating request:", err) return } req.Header.Set("Content-Type", "application/json") req.Header.Set("Authorization", "Bearer "+apiKey) client := &http.Client{} resp, err := client.Do(req) if err != nil { fmt.Println("Request error:", err) return } defer resp.Body.Close() body, err := io.ReadAll(resp.Body) if err != nil { fmt.Println("Error reading response:", err) return } fmt.Println("Status:", resp.Status) fmt.Println("Response:") fmt.Println(string(body)) } ``` ```php "; $payload = ["queryId" => "c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64"]; $ch = curl_init($url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_HTTPHEADER, [ 'Content-Type: application/json', "Authorization: Bearer $apiKey" ]); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($payload)); $response = curl_exec($ch); $httpStatus = curl_getinfo($ch, CURLINFO_HTTP_CODE); curl_close($ch); echo "Status: $httpStatus\n"; echo "Response: $response\n"; ?> ``` ```java import java.io.*; import java.net.HttpURLConnection; import java.net.URL; public class CoralogixQuery { public static void main(String[] args) throws Exception { String endpoint = "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/data"; String apiKey = ""; URL url = new URL(endpoint); HttpURLConnection con = (HttpURLConnection) url.openConnection(); con.setRequestMethod("POST"); con.setRequestProperty("Content-Type", "application/json"); con.setRequestProperty("Authorization", "Bearer " + apiKey); con.setDoOutput(true); String payload = "{\"queryId\": \"c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64\"}"; try (OutputStream os = con.getOutputStream()) { byte[] input = payload.getBytes("utf-8"); os.write(input, 0, input.length); } int status = con.getResponseCode(); InputStream responseStream = (status < 400) ? con.getInputStream() : con.getErrorStream(); BufferedReader in = new BufferedReader(new InputStreamReader(responseStream)); String line; StringBuilder response = new StringBuilder(); while ((line = in.readLine()) != null) { response.append(line.trim()); } in.close(); System.out.println("Status: " + status); System.out.println("Response: " + response.toString()); } } ``` ```csharp using System; using System.Net.Http; using System.Text; using System.Threading.Tasks; class Program { static async Task Main() { var client = new HttpClient(); var url = "https://api.[[DOMAIN_VALUE]]/api/v1/dataprime/background-query/data"; var apiKey = ""; var payload = "{\"queryId\": \"c80fa3e7-fa5b-441a-b0d4-acdbdbe36e64\"}" var content = new StringContent(payload, Encoding.UTF8, "application/json"); client.DefaultRequestHeaders.Add("Authorization", $"Bearer {apiKey}"); var response = await client.PostAsync(url, content); var result = await response.Content.ReadAsStringAsync(); Console.WriteLine($"Status: {(int)response.StatusCode}"); Console.WriteLine("Response:"); Console.WriteLine(result); } } ``` ### Response **Status:** `200 OK` **Content-Type:** `application/x-ndjson` (newline-delimited JSON) **Body format:** A stream of one or more JSON objects, each on its own line. Every object contains a `response.results.results` array with one or more rows (`metadata`, `labels`, `userData`). Clients should read and parse the body **line-by-line** until EOF. ```http Content-Type: application/x-ndjson {"response":{"results":{"results":[{"metadata":[...],"labels":[...],"userData":"..."}]}}} {"response":{"results":{"results":[{"metadata":[...],"labels":[...],"userData":"..."}]}}} ... ``` > Notes > > - Each **line** is a complete JSON object; do not attempt `response.json()` on the entire body. > - Large result sets may arrive as multiple lines (batches). > - If you need to present or store all results, accumulate parsed lines into a single list. ## Response handling The `Content-Type` representation header indicates the original media type of the resource before any content encoding is applied for transmission. In responses, the `Content-Type` header informs the client of the actual content type of the returned data. Data endpoint results are returned in batches, with each batch containing multiple rows formatted as Newline Delimited JSON (NDJSON). Other types of responses are returned in standard JSON format. ### Failed requests The general format guidelines are displayed when the accompanying status code is returned. | Status code | Description | | ----------- | ----------- | | 200 | No Error | | 400 | Bad Request | | 403 | Forbidden | A `400` error typically means the user made a malformed request. A `403` error typically means that there was problem with the API key. See your [API keys page](https://coralogix.com/docs/user-guides/account-management/api-keys/api-keys/index.md) or administrator for more information. ### Warnings The `Warning` response contains information about possible problems with the status of the message. More than one `Warning` header may appear in a response. `Warning` responses can be applied to any message, before or after query results. | **Warning type** | **Description** | | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | CompileWarning | Warning of potential compilation error in your query. In the event of a compilation failure, you will receive an error response. | | TimeRangeWarning | When the time frame for your query has been built incorrectly or exceeds internal limits | | NumberOfResultsLimitWarning | When the number of query results exceeds internal limits | | BytesScannedLimitWarning | When the number of bytes returned in query results exceeds internal limits | | DeprecationWarning | When a value in your query is changed or deprecated incorrectly | ## Limitations When a limit is breached, a **warning message** is displayed. See the limitations page for a comprehensive list of [direct query limitations](https://coralogix.com/docs/dataprime/API/limitations/#direct-query-limitations) and [background query limitations](https://coralogix.com/docs/dataprime/API/limitations/#background-query-limitations). ## Additional resources | | | | ------------------- | ------------------------------------------------------------------------------------------- | | Coralogix Endpoints | [Coralogix Endpoints](https://coralogix.com/docs/integrations/coralogix-endpoints/index.md) |