Analyzing security event logs

The solution to this exercise requires the installation of Apache Spark and Rumble.

• Read the instructions to install Rumble: Installing Spark.
• Download Apache Spark.
• Download Rumble (such as spark-rumble-1.0.jar).

1. Download Apache Spark using a browser that runs inside the VM. Per instructions, move the downloaded file to /usr/local/bin by executing the following command.

   tar zxvf spark-2.4.4-bin-hadoop2.7.tgz sudo mv spark-2.4.4-bin-hadoop2.7 /usr/local/bin

2. Add Apache Spark to your PATH environment variable by executing the following command.

   export SPARK_HOME=/usr/local/bin/spark-2.4.4-bin-hadoop2.7 export PATH=$SPARK_HOME/bin:$PATH

3. Execute the following command to verify the steps.

   cd $SPARK_HOME/bin spark-submit –version cd $HOME spark-submit –version

4. Use a browser running inside the VM to download Rumble.
5. Create a directory from where Rumble is executed (for example $HOME/pega8/rumble).
6. Move the downloaded spark-rumble-1.0.jar into the directory you created.

Define Rule-Utility-Functions in Rule-Utility-Library named Security

Security • LogCustomEventValueGroup() Void

• Parameters: eventType String, outcome String, message String, customFlds ClipboardProperty
• Description: customFlds must be a Text ValueGroup. Call this Function when wanting to log a custom Security Event

Security • NumericTimestamp() String

• Parameters: unit String, allowed values = "h", "m", "s"
• Description: Returns the current timestamp in units of hours, minutes, or seconds as a String

Override FSG-Booking-Work OpenDefaults Extension Point

Enable Geo-Location and Custom Security Events logging

The Chrome browser does not support Geolocation requests unless the server is secured. Chrome allows Geolocation requests within a Linux VM if its IP address is mapped to the name localhost.

Example:

1. Enter the ifconfig IP address as the first entry for localhost.
2. Test by executing ping localhost.
   
   The When rule, pyGeolocationTrackingIsEnabled, must return true.
   
   The When rule applied to Data-Portal displays the following pop-up window after login.

3. Click Allow Location Access.
4. Check for the pxLatitiude and pxLongitude set within the pxRequestor page on your Clipboard.
5. Tip: For best results, use Firefox as your browser.
6. Click Security > Tools > Security > Security Event Configuration to launch the Security Event Configuration landing page.
7. At the bottom of the landing page, click ON to enable custom events.
8. Click Submit.
   
9. Open an Event case for review, and check pyWorkPage on the Clipboard or Trace OpenDefaults to see the values for the customFlds Text Value Group, as shown in the following image.
   
10. Click System > Operations > Logs.
11. To the right of SECURITYEVENT, next click the Text link, click Log Files.
12. If challenged, enter the user name admin and the password admin.
13. Open the SecurityEvent.log file to see whether the customFlds values have been recorded.

Remove Quotes Around Numeric Values (necessary for JSON format)

Use the following steps to remove quotes around numeric values (click here for source).

1. Copy and paste the text that you want the regular expression to examine.
2. Replace each numeric value with (-?[0-9]*\.[0-9]*) if the value can be negative, or ([0-9]*\.[0-9]*) if the value cannot be negative.
3. Use \1, \2, and so on, to specify which parsed value goes where.

Replace newlines within an entire file, not one line at a time

If newlines exist in a file read by a Unix script, each line is processed individually. You must treat the entire file as a single large String, and then replace the new lines within that String (click here for source).

The output of the newline-removing sed command is piped to sed again, this time to replace: }<space>{ with: }<comma>{

Have one file, test2.jq, act as the start of the query including the JSON array left bracket: let $log := [

Have a second file, restof_test1.jq, contain the end of the JSONiq query starting at the JSON array's right bracket.

1. The script initializes a new query file using > test2.jq
2. Then output the back-to-back sed commands using >> test2.jq
3. Then appends restof_test1.jq using >> test2.jq

Next is the remainder of the JSONiq query (restof_test1.jq). The distance between two coordinates is computed in miles (3958.8 is the radius of the earth in miles), and then divided by the time difference between the two log file entries. The timestamp is computed as hours since 1-1-1970 GMT.

The filter condition ensures pairs of rows are only examined when:

1. The eventType is Open Work (see FSG-Booking-Work OpenDefaults override above)
2. The operatorID values within the two records are identical.
3. The latitudes in both rows are > 20 (this is used to ensure that both rows contain geolocation coordinates. Any comparison can suffice).
4. The timestamp in the second row is in the future of the timestamp in the first row (this avoids redundant calculation where the velocity ends up negative).

Send the JSONiq query to Apache Spark as opposed to executing commands using shell mode

Ideally, there is an easier way to send the queryfile content to Apache Spark and Rumble than rather than open the file, select all, copy, and paste into Rumble's command line (click here for source).

1. Instead of: --shell yes
2. Use: --query-path file.jq --output-path results.out
3. In a directory where the Rumble .jar exists on a system where Apache Spark is installed, execute the following command:

The example test2.out can be interpreted as:

Two Security Events were recorded in close succession. To have traversed the distance of 31.6 miles (50.9 km) within the time between when those two Security Events were logged requires someone to have traveled at a speed of 2634 miles per hour (4329 km/hr).