How-To: Reassemble a UDP Syslog Event

The UDP Syslog Event Reassembly feature in the LogScale Collector addresses the challenge of fragmented syslog messages by reconstructing related UDP syslog events into a single, coherent event

Message Structure

Each fragmented message contains:

Timestamp and other standard syslog headers
A shared message ID (e.g., 0002848781, 0002848782, 0002848783)
Total fragment count (e.g., 2, 3, 2)
Fragment index (starting from 0)
Partial content of the full message

How it works

The collector groups incoming message fragments by their shared ID into an internal buffer, tracking received fragments against the total expected count. When all fragments for a message have been received, they are combined into a single event. This combined event is then entered into the LogScale collector queue for further processing.

The original syslog headers are preserved in the reassembled event. Additionally, the message ID and fragment count are added as associated fields to the event for reference and troubleshooting purposes.

Error Handling

How different errors are handled.

If an event doesn't match the expected pattern, the event is passed through unaltered.
If a fragment index exceeds the total fragment count, the already received part of the event is submitted.
If the total fragment count exceeds a limit (default 20), the event is passed through unaltered.
If all fragments haven't been received within a window of (default 1000) events, the already received part of the event is submitted.
If the internal buffer (default 16 MB) is overrun, the oldest partially received event is submitted.

Note

When an event is partially reassembled due to any of the above conditions, a [email protected] is added to the event with a message describing why the event was only partially reassembled.

Configuration

The feature is configured by adding the joining field under a syslog source in the collector configuration file. The following fields can be specified under joining:

sources:
        my_syslog:
          type: syslog
          mode: udp
          sink: my_logscale
          joining:
            type: cisco-ise
            # Optional: Specify the maximum amount of bytes stored in the joining buffer.
            # bufferSize: 16777216
            # Optional: Specify the maximum number of events that can be received before evicting an incomplete event.
            # windowSize: 1000
            # Optional: Specify the maximum number of UDP events that can be combined into a single event.
            # eventLimit: 20

Example

<181>Oct2 16:05:00 [HOSTNAME]_RADIUS_Accounting 0002848782 3 0 2024-10-02 16:05:00.677+00:00 0092300347 3002 NOTICERadius-Accounting: RADIUS Accounting watchdog update, ConfigVersionId=75, DeviceIP Address=[INTERNAL_IP], UserName=[USER1], NetworkDeviceName=[FIREWALL01], User-Name=[USER1], NAS-IP-Address=[INTERNAL_IP], NAS-Port=3012980736, Service-Type=Framed, Framed-Protocol=PPP, Framed-IP-Address=[INTERNAL_IP], Class=CACS:[HASH_VALUE]:[HOSTNAME]/515384413/2817912,Called-Station-ID=[PUBLIC_IP], Calling-Station-ID=[PUBLIC_IP], Acct-Status-Type=Interim-Update, Acct-Delay-Time=0, Acct-Input-Octets=10022011, Acct-Output-Octets=86684367, Acct-Session-Id=[SESSION_ID], Acct-Authentic=RADIUS, Acct-Session-Time=3628, Acct-Input-Packets=81690, Acct-Output-Packets=131153, NAS-Port-Type=Virtual, Tunnel-Client-Endpoint=(tag=0) [PUBLIC_IP], cisco-av-pair=mdm-tlv=device-platform=win, cisco-av-pair=mdm-tlv=device-mac=[MAC_ADDRESS], cisco-av-pair=audit-session-id=[HASH_VALUE]

<181>Oct2 16:05:00 [HOSTNAME]_RADIUS_Accounting 0002848782 3 1 cisco-av-pair=mdm-tlv=device-platform-version=10.0.22621, cisco-av-pair=mdm-tlv=device-public-mac=[MAC_ADDRESS], cisco-av-pair=mdm-tlv=device-type=[DEVICE_MODEL], cisco-av-pair=mdm-tlv=ac-user-agent=AnyConnect Windows5.2.1.00, cisco-av-pair=mdm-tlv=device-uid-global=[DEVICE_UID], cisco-av-pair=mdm-tlv=device-uid=[DEVICE_UID], CVPN3000/ASA/PIX7x-Tunnel-Group-Name=[VPN_GROUP], CVPN3000/ASA/PIX7x-Client-Type=2, CVPN3000/ASA/PIX7x-Session-Type=1, CVPN3000/ASA/PIX7x-Session-Subtype=3, AcsSessionID=[HOSTNAME]/515384413/2818247, SelectedAccessService=PAP_ASCII, RequestLatency=4, Step=11004, Step=11017, Step=15049, Step=15008, Step=22085, Step=11005, NetworkDeviceGroups=[LOCATION], NetworkDeviceGroups=[DEVICE_GROUP], NetworkDeviceGroups=[DEPLOYMENT_PHASE]

<181>Oct2 16:05:00 [HOSTNAME]_RADIUS_Accounting 0002848782 3 2 CPMSessionID=[SESSION_ID], TotalAuthenLatency=4, ClientLatency=0, Deployment_Phase=[DEPLOYMENT_PHASE], ModelName=[DEVICE_MODEL], Software Version=x.y, Network DeviceProfile=[VENDOR], Location=[LOCATION], Device Type=[DEVICE_GROUP]

Knowledge Base