Lookup Files

Security Requirements and Controls

Lookup files are used to add additional context to data, enabling you to attach or replace text from events recorded in a repository when searched.

To add a lookup file, create or import a CSV (comma-separated value) or JSON file and upload it to the repository or view. An overview table allows for searching and filtering, to easily find and manage the available files.

Files View

Figure 33. Files View


These files can be used together with query functions to provide lookups and matching using the match() function.

The feature also works with the readFile() function for reading a file which is used as data input for your query.

The following operations are available:

For information on how Lookup files interact with the rest of the system, see Lookup Files Operations.

Supported File Types and Formats

LogScale supports two different file formats for uploaded lookup files. JSON and CSV.

  CSV Files JSON Files
Viewable within LogScale UI Yes No
Editable within LogScale UI Yes No
File Size Limit 200MB 100MB

Lookup Files using CSV Format

When using CSV for lookup files, the following rules apply:

  • Individual fields should be separated by a comma (,)

  • Whitespace is always included in the imported fields, the input takes the literal contents split by the comma character.

  • Fields can optionally be quoted by double quotes, for example to include commas in the imported values.

  • The first line of the CSV is interpreted as the column header and can be used as the field name when looking up values with functions like match().

For example, the CSV file:

csv
number,code,description
17,udp,UDP
80,http,HTTP Service
ip,"Internet Protocol, pseudo protocol"

Would be interpreted as:

numbercodedescription
17udpUDP
80httpHTTP Service
ipInternet Protocol, pseudo protocol 

CSV files can be viewed within the Files interface to confirm how the information has been interpreted.

Lookup Files using JSON Format

When using JSON files, two different formats are supported, object-based and array-based.

Important

Once uploaded, JSON files cannot be viewed or updated. They can be exported to confirm the file format.

  • Object-based

    In the object-based format, the JSON should be formatted as a hash or associative array, with a single key and corresponding object. For example:

    json
    {
  "1": { "name": "chr" },
  "2": { "name": "krab" },
  "4": { "name": "pmm" },
  "7": { "name": "mgr" }
}

    When performing a lookup, match() will return the object (as an event with multiple fields), based on the matching key.

  • Array-based

    In the array-based format, the JSON should be formatted as an array of objects. In this model, the keys for each individual object become fields that can be matched when performing a lookup. For example, in the file:

    json
    [
  { "userid": "1", "name": "chr" },
  { "userid": "2", "name": "krab" },
  { "userid": "4", "name": "pmm" },
  { "userid": "7", "name": "mgr" }
]

    The userid and name fields in the JSON object can be used to lookup and return other key/value pairs as event fields/values. For example, the fragment:

    logscale Syntax
    ...
| match(file="long.json",field=codename,column="name")

    Would return the userid field for objects within the lookup file array.

For both formats, the following common apply:

  • JSON must be formatted in strict notation format. This requires no trailing commas (where there is no additional value),

  • Individual keys and values should be quoted, even as a number.

Important

Nested objects, that is an object within the returned object, are not supported. For example:

Invalid Example for Demonstration - DO NOT USE
json
{
  "1": { "name": "chr", "roles": { "user" : true }},
  "2": { "name": "krab" },
  "4": { "name": "pmm" },
  "7": { "name": "mgr" }
}

Would return only the simple field, name when used with match(); the remainder of the embedded object would be not be returned or included in the events. LogScale does not reject files in this format.

Create a File

Security Requirements and Controls

  • Create file permission

  1. Click Files → + New File → Create New.

  2. Specify a name for the file and then select either + Empty File to create an empty file to populate or From Package to use a template from a previously installed package.

  3. Click Create file.

  4. If you've created an empty file, click + to add rows and columns.

  5. Click Save to save the changes.

If you have many changes to make, editing a data table through the Files interface page can be tedious: click Export and then edit the table in a spreadsheet program or a simple text editor.

Note

Files larger than 100 MB cannot be viewed in the UI.

Create New CSV File

Figure 34. Create New CSV File


When a file is referenced in a query, a tab is shown in the Search page bearing the same name of the file. This file tab will display the file content as a Table widget. Alternatively, if the file cannot be queried, a download link will be presented instead. For example, executing the query:

logscale
groupBy([status])
| match(file="status_codes.csv", column="code", field="status", include=name)

will show a file table named status_codes.csv:

File Tab in Search View

Figure 35. File Tab in Search View


Upload Files

  1. Go to the Files interface → + New file → Import files.

  2. Drag and drop your file or browse for the file to upload. You can import multiple files at once.

    You can upload a CSV file containing text like what you see below, which is essentially a lookup table that you can use for labels or value lookups.

    csv
    userid,ip,username,region
1,"212.12.31.23","pete","EU"
2,"212.12.31.231","bob","EU"
3,"98.12.31.21","anders","EU"
4,"121.12.31.23","jeff","US"
5,"82.12.31.23","ted","AU"
6,"62.12.31.23","annie","US"
7,"122.12.31.23","joe","CH"
8,"112.11.11.21","alice","CH"
9,"212.112.131.22","admin","RU"
10,"212.12.31.23","wendy","EU"

    Once it has been uploaded, it will look like what you see in figure below.

    Import CSV File

    Figure 36. Import CSV File


    Typically, the content is used within the match() to lookup fixed reference information. Notice that the values are in quotes, except for the ones for userid, which are integers. See the Lookup API reference page for more information on this topic.

  3. Once created of uploaded, the file can be edited and updated withing the user interface. Additional columns and rows can be added to the file using the + button. Clicking the tiny information icon next to the file name displays metadata info about the file (created by, time it was created, etc.)

    Important

    Only CSV files can be edited once uploaded within the user interface.

  4. Once you have finished editing, click Save, or click Export if you wish to download the edited file.

Export or Delete a File

Security Requirements and Controls

  • Delete file permission

Files can be managed by clicking the menu icon next to each file. You can either export or delete a file:

Manage CSV Files

Figure 37. Manage CSV Files


Warning

Deleting a file that is actively used by live queries will stop those queries.

Permissions for files

Security Requirements and Controls

Sometimes you might want to collaborate with another user on a file, but that user does not have permission to files in the view. If you have permissions to do so, you can grant permissions to that user to edit and delete a particular file in a view. For more information about asset permissions, see Asset permissions.

If you do not have Change user access permission on the repository, you will see a list of users only (no groups) that already have at least Read permissions on the repository. You can select from these users and give them more permissions (up to the same permissions you have).

To grant access to edit or delete a file to another user or group:

Asset creator/Regular user

The creator of an asset and regular users can share the same permissions that they have to the asset with users who already have read access to the view. You cannot share access with users who do not have read access to the view. You cannot share access with groups at all.

  1. Click ⋮ next to the file you want to share and select Asset sharing.

  2. In the Users and groups with access window you see users who currently have access to the file and what access they have.

  3. Click Share file.

  4. Click to select the user to get additional permissions. Note that you can only see users who already have read permission to the view. Click Next.

  5. Select the appropriate permissions to assign the permissions. You can only grant up to the same permissions you have. Click Grant permissions.

You have Change user access permission

With Change user access permission, you can grant permission to users, including read permission if the user does not have that, and permissions that you do not have yourself. You can also see groups and group members and what permissions they have in the Groups tab, but you cannot change the permissions for the group in the Groups tab. To be able to change the permissions directly from the group tab, you must have Change organization permissions permission.

To grant additional permissions to a user that already has read access to the view:

  1. Click ⋮ next to the file you want to share and select Asset sharing.

  2. In the Users and groups with access window you see users and groups who currently have access to the file and what access they have.

  3. Click the button next to the user in the list.

  4. Click to assign the permissions. Click Save changes.

  5. Click Close.

If you have the Change user access permission and you want to share permissions to the file with a user or group not in the list, or you want to give a group that is in the list additional permissions:

  1. Click Share file.

  2. Click to select the group or user who should get additional permissions. Click Next.

  3. Select the appropriate permissions to assign. Be aware of the message that the user or group gets Read access to all assets in the repository automatically when assigning asset permissions for one asset in the repository. Click Next.

  4. Confirm that you understand that you are granting Read access to all assets in the repository by adding the asset permission for the user or group. Click Grant data read access.

  5. Click Grant permissions.

Lookup Files Operations

When using Lookup files and match() functionality, consider the following:

  • Lookup files use server memory proportional to the size of the file on disk; at least as much and typically more. If you have a 1Gb lookup file it will take up at least 1Gb of memory on some, potentially all, hosts within the cluster. This requirement should be taken into account when uploading and sizing the nodes within the cluster.

  • From LogScale v1.108 on, content of the file is shared among all queries that uses match(), that is, the included columns that are common among match() functions can be reused among queries.

  • From 1.117 version on, whenever a file is updated, live queries and alert queries that use that file will seamlessly continue to run with the new updated file, thus making little difference if you have many small files to update or one large file. Since the file is swapped while the query is running, this also means that events can be queried with different versions of the file.

  • From LogScale v1.90, if you have large lookup files, wrap the uses of match() in saved queries rather than use them directly across multiple different queries to ensure you don't accidentally pass slightly different argument in different queries. However, due to an improved reuse of files introduced in LogScale v1.108, this practice is no longer necessary starting from that version.