One of the main problems of existing approaches in anomaly detection in intrusion detection system (IDS) is that IDSs provide only binary detection result: intrusion (attack) or normal. If some attack data or normal data is belonged to boundary, they may be classified wrongly. That is a main cause of high false rates and inaccurate detection rates in IDS. We propose a new approach named Quantitative Intrusion Intensity Assessment (QIIA) that exploits proximity metrics computation so that it provides intrusion (or normal) quantitative intensity value. It is capable of representing how an instance of audit data is proximal to intrusion or normal in a numerical value. This can identify unknown intrusion and normal pattern more accurately. Prior to applying QIIA to audit data, we perform feature selection and parameter optimization of detection models to decrease the overheads to process audit data and to enhance detection rates. Random Forests is used to generate proximity metrics that represent the intrusion intensity (and normal instance intensity) in a numerical way. The numerical value is used to determine whether unknown audit data are intrusion or normal. We carry out several experiments on KDD 1999 dataset and the experimental results show the feasibility of our approach. Copyright © 2012 John Wiley & Sons, Ltd.