Abstract

One crucial thing that hackers always do is gather information about the state of networks by breaking into files and systems that are used in defense models. Threats can get into these platforms. Much information is constantly coming into and going out of systems and people. To find potentially harmful security trends, the Intrusion Detection System (IDS) has to look through this growing amount of data. Our surroundings and choices now make it very hard to quickly and correctly find intrusions. To find people who try to get into a communication system without permission, creating an intrusion detection system (IDS) that uses big data techniques to handle large amounts of complex data is essential. This work uses artificial intelligence (AI), which is aware of a vast amount of data, to make an IDS that is very good at dealing with these problems. The study created a unique structure for the Long Short-Term Memory (LSTM) method, which can find complex links and long-lasting patterns in arriving at network traffic data. By using this method, the system can successfully lower the number of false warnings and improve the accuracy of the IDS that was created. Big Data Analysis (BDA) could make the AI methods discussed in this study more useful. These methods are currently slow because they are very complicated. Using these techniques makes running the complicated model go more quickly. The researchers used the tool on the Spark platform for in-depth studies. The NSL-KDD database was used to train for the tests. The results show that the algorithm is better than other IDS systems regarding how often it finds problems, how usually it sets off fake alarms, how accurate it is, how efficiently it works, and how long it takes to train.