Log forwarding to any destination
AppLogs usually ingests, parses, and indexes the logs so you can search and analyze them from the monitoring console. Log forwarding builds on that: It can stream a copy of the logs to either a cloud storage destination or an HTTP endpoint that you or your SIEM solutions controls. This enables you to retain, archive, or process them in your own environment.
Once a forwarding configuration is active, every matching log batch is delivered to your configured destination in the supported format, based on the configured batch settings, and independently of the AppLogs search and storage retention.
Supported destinations
Storage
SIEM and streaming
Managing log forwarding configurations
To view the created configurations, go to Admin > Third-Party Integrations > Log Forwarding. This will list all the configured log forwarding integrations. The details are listed in the below content format:
| Parameter | Description |
|---|---|
| Name | Displays the name given to the configuration. |
| Destination | Displays the supported destination the logs are forwarded to. |
| Log Types | Displays the type of logs included in the configuration, 100+ default logs or custom log types. |
| Frequency | Displays the forwarding frequency to the destination. |
| Status | Displays the configuration status, whether the configuration is Active or Inactive. |
| Last Upload | Displays the last time the logs were forwarded to the destination. |
The hamburger icon 
is displayed next to each configured name. Click the icon to access the following actions:
- Disable forwarding to disable log forwarding.
- Enable forwarding to enable log forwarding. Both the actions are updated in the Status column.
- Delete to clear the created log forwarding destination.
Filtering logs before forwarding
By default, all logs of the selected log type are forwarded. To forward only a subset, navigate to Admin > Third-Party Integrations > Log Forwarding, open the destination you have configured, and enter a Filter Query using the same search syntax you use on the AppLogs search page. Only log records that match the query are delivered; the rest remain searchable in AppLogs as usual, but aren't forwarded.
For example, to forward only error-level records, you might use a query such as:
logtype="zylker logs" severity = "ERROR"
Batching and delivery options
Forwarded logs are grouped in batches before they're written to reduce the number of destination objects and helps manage storage costs. Navigate to Admin > Third-Party Integrations > Log Forwarding, then open the destination you've configured. There are two settings control batching:
- Batch Size: The maximum number of records per object. There are 1000 records by default.
- Batch Interval: The maximum time a batch is held before it's written, in seconds. It is 60 seconds by default.
A batch is written as soon as either threshold is reached.
Larger batches reduce the number of destination objects, which can lower storage and request costs. However, they also increase log delivery latency. Select batch settings that best meet your latency and cost requirements.
Testing the connection
Before saving the configuration, or any time afterward, to verify that the destination is reachable and the configured credentials have the required permissions.
- Navigate to Admin > Third-Party Integrations > Log Forwarding, and open the destination you've configured.
- Click Test Connection.
- Site24x7 writes a small test object to the destination and reports the result:
- Success: The destination is reachable, and the configured credentials have permission to write objects.
- Failure: The error message identifies the cause, such as insufficient permissions, an incorrect bucket or region, or an unreachable endpoint. Resolve the issue and run the test again.
A successful test only verifies write access at the time of testing. If the credentials are rotated or permissions change later, log forwarding may fail. Re-test the connection and monitor the forwarding status after making any changes.
Output format and file organization
Each batch is written as a single gzip-compressed NDJSON object (Content-Type: application/gzip; Content-Encoding intentionally unset so the bytes stay compressed). The body format is configurable as NDJSON (one JSON object per line, default), a JSON array, or CSV, with one object generated per batch.
For object storage destinations, files are organized by log type and ingest time using the following path structure:
{path-prefix}/{logType}/{YYYY}/{MM}/{DD}/{HH}/{batch-id}.{format}.gz
The batch ID is derived from the batch contents, so the same batch always maps to the same file name.
The {format} is json, csv, or the application extension for the selected format.
For webhook destinations, the same log records are sent in JSON array, either compressed or uncompressed in the request body.
Log delivery and duplicate handling
Log forwarding provides at-least-once delivery, which ensures that every log record is delivered to the destination at least once. After a rare interruption, such as after a network interruption, a batch may be delivered more than once.
Because object names are derived from the batch contents (a SHA-256 digest), a batch re-delivered with the same contents overwrites the existing object with the same name instead of creating a duplicate.
If a batch is re-grouped during a retry, its digest and corresponding object name may change, which can result in a duplicate object.
To simplify duplicate handling:
- Use a destination bucket or container without object versioning, so that a re-delivered batch replaces the existing object instead of creating a new version.
- Each forwarded record includes a unique
s247_forward_idfield, that can be used to remove duplication records.. Since IDs are generated separately for each destination, check and remove duplicate records within each destination's data independently. The request body is sent as an uncompressed JSON array.
Licensing and usage limits
- You can create up to ten forwarding configurations per account.
- Forwarded log volume counts toward your log forwarding usage. You can review it on the Log Usage Summary page.
- Log forwarding operates independently of AppLogs search. Deleting a forwarding configuration does not affect log collection or search.
- 10GB of log forwarding is one dollar per month and is available as an add-on for all plans