Disc Consulting Enterprises (DCE) has identified some potentially suspicious attacks on their network and computer systems. The attacks are thought to be a new type of attack from a skilled threat actor.

Assignment 2

Machine Learning Modelling

Background

Early Incident Identification

Disc Consulting Enterprises (DCE) has identified some potentially suspicious attacks on their network and computer systems. The attacks are thought to be a new type of attack from a skilled threat actor. To date, the attacks have only been identified 'after the fact' by examining post-exploitation activities of the attacker on compromised systems. Unfortunately, the attackers are skilled enough to evade detection and the exact mechanisms of their exploits have not been identified.

The incident response team, including IT services, security operations, security architecture, risk management, the CISO (Chief Information Security Officer), and the CTO (Chief Technology Officer) have been meeting regularly to determine next steps.

It has been suggested that the security architecture and operations teams could try to implement some real-time threat detection using machine learning models that build on earlier consultancy your firm has completed (i.e., building upon your Assessment 1 work).

Data description

The data have already been provided (in Assessment 1), and the ML team (you) have undertaken some initial cleaning and analysis.

Things to keep in mind:

Each event record is a snapshot triggered by an individual network 'packet'. The exact triggering conditions for the snapshot are unknown. But it is known that multiple packets are exchanged in a 'TCP conversation' between the source and the target before an event is triggered and a record created. It is also known that each event record is anomalous in some way (the SIEM logs many events that

□ maybe suspicious).

□ The malicious events account for a very small amount of data. As such, your

training needs to consider the "imbalanced” data and the effect these data may have on accuracy (both specificity and sensitivity).

A very small proportion of the data are known to be corrupted by their source systems and some data are incomplete or incorrectly tagged. The incident response team indicated this is likely to be less than a few hundred records.

The needle in the haystack

The data were gathered over a period of time and processed by several systems in order to associate specific events with confirmed malicious activities. However, the number of confirmed malicious events was very low, with these events accounting for less than 1% of all logged network events.

Because the events associated with malicious traffic are quite rare, rate of 'false negatives’ and 'false positives’ are important.

Scenario

Following the meetings of the security incident response team, it has been decided to try to make an 'early warning’ system that extends the functionality of their current SIEM. It has been proposed that DCE engage 3^rdparty developers to create a 'smart detection plugin’ for the SIEM.

The goal is to have a plug-in that would extract data from real-time network events, send it to an external system (your R script) and receive a classification in return. However, for the plugin to be effective it must consider the alert-fatigue experienced by security operations teams as excessive false-positives can lead to the team ignoring real incidents. But, because the impact of a successful attack is very high, false negatives could result in attackers overtaking the whole network.

You job

Your job is to develop the detection algorithms that will provide the most accurate incident detection. You do not need to concern yourself about the specifics of the SIEM plugin or software integration, i.e., your task is to focus on accurate classification of malicious events using R.

You are to test and evaluate two machine learning algorithms to determine which supervised learning model is best for the task as described.

Task

You are to import and clean the same MLData2023.csv, that was used in the previous assignment. Then run, tune and evaluate two supervised ML algorithms (each with two types of training data) to identify the most accurate way of classifying malicious events.

Part 1 - General data preparation and cleaning

a) Import the MLData2023.csv into R Studio. This version is the same as Assignment 1.

b) Write the appropriate code in R Studio to prepare and clean the MLData2023 dataset as follows:

i. Clean the whole dataset based on what you have suggested / feedback received for Assignment 1.

ii. Filter the data to only include cases labelled with Class = 0 or 1.

iii. For the feature Operating.System, merge the three Windows categories together to form a new category, say Windows_All. Furthermore, merge iOS, Linux (Unknown), and Other to form the new category named Others. Hint: use the forcats:: fct_collapse(.) function.

iv. Similarly, for the feature Connection.State, merge INVALID, NEW and RELATED for form the new category named Others.

v. Select only the complete cases using the na.omit(.) function, and name the dataset MLData2023_cleaned.

Briefly outline the preparation and cleaning process in your report and why you believe the above steps were necessary.

c) Use the code below to generated two training datasets (one unbalanced mydata.ub.train and one balanced mydata.b.train) along with the testing set (mydata.test). Make sure you enter your student ID into the command set.seed(.).

#  Separate samples of non-maLicious and malicious events

dat.class0 <- MLData2023_cleaned %>% filter(Class == 0) # non-maLicious dat.classl <- MLData2023_cleaned %>% filter(ciass == l) # malicious

#  Randomly select 19800 non-maLicious and 200 malicious samples, then combine them to form the training samples set.seed(Enter your student ID)

rows.train0 <- sample(1:nrow(dat.class0), size = 19800, replace = FALSE) rows.trainl <- sample(1:nrow(dat.class1), size = 200,replace = FALSE)

#  Your 20000 unbalanced training samples

train.class0 <- dat.class0[rows.train0,] # Non-malicious samples train.class1 <- dat.class1[rows.train1,] # Malicious samples mydata.ub.train <- rbind(train.class0, train.class1) mydata.ub.train <- mydata.ub.train %>%

mutate(Class = factor(Class, labels = c("NonMal","Mal")))

#  Your 39600 balanced training samples, i.e. 19800 non-malicious and malicious sam ples each. set.seed(123)

train.class1_2 <- train.class1[sample(1:nrow(train.class1), size = 19800, replace = TRUE),] mydata.b.train <- rbind(train.class0, train.class1_2) mydata.b.train <- mydata.b.train %>%                                                                                                   mutate(Class =

factor(Class, labels = c("NonMal","Mal")))

#  Your testing samples

test.class0 <- dat.class0[-rows.train0,] test.class1 <- dat.class1[-rows.train1,] mydata.test <- rbind(test.class0, test.class1) mydata.test <- mydata.test %>%

mutate(Class = factor(Class, labels = c("NonMal","Mal")))

Note that in the master data set, the percentage of malicious events is less than 1%. This distribution is roughly represented by the unbalanced data. The balanced data is generated based on up-sampling of the minority class using bootstrapping. The idea here is to ensure the trained model is not biased towards the majority class, i.e. non-malicious event.

Part 2 - Compare the performances of different ML algorithms

a) Randomly select two supervised learning modelling algorithms to test against one another by running the following code. Make sure you enter your student ID into the command set.seed(.). Your 2 ML approaches are given by myModels.

set.seed(Enter your student ID)

models.listl <- c("Logistic Ridge "Logistic LASSO "Logistic E] models.list2 <- c("Classification "Bagging Tree", "Random Forest") myModels <- c(sample(models.list1, sample(models.list2, myModels %>%

data.frame

For each of your two ML modelling approaches, you will need to:

b) Run the ML algorithm in R on the two training sets with Class as the outcome variable.

c) Perform hyperparameter tuning to optimise the model:

• Outline your hyperparameter tuning/searching strategy for each of the ML modelling approaches. Report on the search range(s) for hyperparameter tuning, which fc-fold CV was used, and the number of repeated CVs (if applicable), and the final optimal tuning parameter values and relevant CV statistics (i.e. CV results, tables and plots), where appropriate.

• If your selected tree model is Bagging, you must tune the nbagg, cp and minsplit hyperparameters, with at least 3 values for each.

• If your selected tree model is Random Forest, you must tune the num.trees and mtry hyperparameters, with at least 3 values for each.

• Be sure to set the randomisation seed using your student ID.

d) Evaluate the predictive performance of your two ML models, derived from the balanced and unbalanced training sets, on the testing set. Provide the confusion matrices and report and describe the following measures in the context of the project:

• False positive rate

• False negative rate

• Overall Accuracy

• Precision

• Recall

• F-score

For the precision, recall and F-score metrics, you will need to do a bit of research as to how they can be calculated. Make sure you define each of the above metrics in the context of the study.

e)

Provide a brief statement on your final recommended model and why you chose it. Parsimony, and to a lesser extent, interpretability maybe taken into account if

the decision is close. You may outline your penalised model if it helps with your

argument.

What to submit

Gather your findings into a report (maximum of 5 pages) and citing relevant sources, if necessary.

Present how and why the data was 'cleaned and prepared’, how the ML models were tuned and provide the relevant CV results. Lastly, present how they performed to each other in both the unbalanced and balanced scenarios. You may use graphs, tables and images where appropriate to help your reader understand your findings. All tables and figures should be appropriately captioned, and referenced in-text.

Make a final recommendation on which ML modelling approach is the best for this task.

Your final report should look professional, include appropriate headings and subheadings, should cite facts and reference source materials in APA-7^th format.

Your submission must include the following:

• Your report (5 pages or less, excluding cover/contents/reference/appendix page). The report must be submitted through TURNITIN and checked for originality.

• A copy of your R code, and three csv files corresponding to your two training sets and a testing set. The R code and data sets are to be submitted separately via another submission link.

Note that no marks will be given if the results you have provided cannot be confirmed by your code. Furthermore, all pages exceeding the 5-page limit will not be read or examined.

Marking Criteria

Academic Misconduct

Academic misconduct, which includes but is not limited to, plagiarism; unauthorised collaboration; cheating in examinations; theft of other student's work; collusion; inadequate and incorrect referencing; will be dealt with in accordance with the Rule for Academic Misconduct (including Plagiarism) Policy. Ensure that you are familiar with the Academic Misconduct Rules.