Analyzing security event logs

The solution to this exercise requires the installation of Apache Spark and Rumble. In the following steps, you must: 

• Read the installation instructions for Rumble on the RumbleDB documentation website.
• Download Apache Spark.
• Download the latest Rumble JAR file from GitHub.

1. Download Apache Spark using a browser that runs inside the VM. Move the downloaded file to /usr/local/bin by running the following command:

   tar zxvf spark-2.4.4-bin-hadoop2.7.tgz sudo mv spark-2.4.4-bin-hadoop2.7 /usr/local/bin

2. Add Apache Spark to your PATH environment variable by running the following command:

   export SPARK_HOME=/usr/local/bin/spark-2.4.4-bin-hadoop2.7 export PATH=$SPARK_HOME/bin:$PATH

3. Run the following command to verify the steps:

   cd $SPARK_HOME/bin spark-submit –version cd $HOME spark-submit –version

4. Use a browser running inside the VM to download Rumble.
5. Create a directory from where you can run Rumble. For example, $HOME/pega8/rumble.
6. Move the spark-rumble-x.x.jar file that you downloaded into the directory that you created.

Define the Rule-Utility-Functions in a Rule-Utility-Library named Security

Security • LogCustomEventValueGroup() Void

• Parameters: eventType String, outcome String, message String, customFlds ClipboardProperty
• Description: customFlds must be a Text ValueGroup. Call this Function when you want to log a custom Security Event

Security • NumericTimestamp() String

• Parameters: unit String, allowed values = "h", "m", "s"
• Description: Returns the current timestamp in units of hours, minutes, or seconds as a String

Override the MDC-DS-Work OpenDefaults Extension Point

Enable Geo-Location and Custom Security Events logging

The Chrome browser does not support Geolocation requests unless the server is secure. Chrome allows Geolocation requests within a Linux VM if its IP address is mapped to the name localhost. For example:

1. Enter the ifconfig IP address as the first entry for localhost.
2. Test by entering ping localhost.
   
   The pyGeolocationTrackingIsEnabled When rule must return true.
   
   The When rule applied to Data-Portal displays the following window after you log in:

3. Click Allow Location Access.
4. Check for the pxLatitiude and pxLongitude set within the pxRequestor page on your Clipboard.
5. Click Security > Tools > Security > Security Event Configuration to launch the Security Event Configuration landing page.
6. At the bottom of the landing page, click ON to enable custom events.
7. Click Submit.
   
8. Open an Event case for review, and check pyWorkPage on the Clipboard or Trace OpenDefaults to see the values for the customFlds Text Value Group, as shown in the following figure:
   
9. Click System > Operations > Logs.
10. To the right of SECURITYEVENT, click Log Files.
11. If challenged, enter the user name admin and the password admin.
12. Open the SecurityEvent.log file to see whether the customFlds values have been recorded.

Remove Quotes Around Numeric Values (necessary for JSON format)

Use the following steps to remove quotes around numeric values:

1. Copy and paste the text that you want the regular expression to examine.
2. Replace each numeric value with (-?[0-9]*\.[0-9]*) if the value can be negative, or ([0-9]*\.[0-9]*) if the value cannot be negative.
3. Use \1, \2, and so on, to specify which parsed value goes where.

Replace newlines within an entire file, not one line at a time

If newlines exist in a file read by a Unix script, each line is processed individually. You must treat the entire file as a single large String, and then replace the new lines within that String.

The output of the newline-removing sed command is piped to sed again, this time to replace: }<space>{ with: }<comma>{

Have one file, test2.jq, act as the start of the query including the JSON array left bracket: let $log := [

Have a second file, restof_test1.jq, contain the end of the JSONiq query starting at the JSON array's right bracket.

1. The script initializes a new query file using > test2.jq
2. Output the back-to-back sed commands using >> test2.jq
3. Append the content of restof_test1.jq to test2.jq by using the following commands:

Next is the remainder of the JSONiq query (restof_test1.jq). The distance between two coordinates is computed in miles (3958.8 is the radius of the earth in miles), and then divided by the time difference between the two log file entries. The timestamp is computed as hours since 1-1-1970 GMT.

The filter condition ensures pairs of rows are only examined when:

1. The eventType is Open Work (see the MDC-DS-Work OpenDefaults override above)
2. The operatorID values within the two records are identical.
3. The latitudes in both rows are > 20 (this is used to ensure that both rows contain geolocation coordinates. Any comparison can do).
4. The timestamp in the second row is in the future of the timestamp in the first row (this avoids redundant calculation where the velocity ends up as a negative).

Send the JSONiq query to Apache Spark instead of running commands using shell mode

Ideally, there is an easier way to send the queryfile content to Apache Spark and Rumble, rather than opening the file, selecting all, copying and pasting into Rumble's command line.

1. Instead of: --shell yes
2. Use: --query-path file.jq --output-path results.out
3. In a directory where the Rumble JAR file exists, on a system where Apache Spark is installed, run the following command:

The test2.out example can be interpreted as follows:

Two Security Events were recorded in close succession. To have traversed a distance of 31.6 miles (50.9 km) within the time between when those two Security Events were logged requires someone to have traveled at a speed of 2,634 miles per hour (4,329 km/hr).