Issues in Detection of AI-Generated Source Code

Bukhari, Sufiyan Ahmed

Issues in Detection of AI-Generated Source Code

dc.contributor.advisor	De Carli, Lorenzo
dc.contributor.author	Bukhari, Sufiyan Ahmed
dc.contributor.committeemember	Tan, Peng Seng Benjamin
dc.contributor.committeemember	Abdel Latif, Ahmad Hazzaa Khader
dc.date	2024-11
dc.date.accessioned	2024-09-19T14:40:19Z
dc.date.available	2024-09-19T14:40:19Z
dc.date.issued	2024-09-17
dc.description.abstract	AI coding assistants hold the promise of revolutionizing software development, but they also introduce new risks. The underlying large language models (LLMs) can generate faulty or insecure code either unintentionally or through malicious attacks. Additionally, there's a potential for AI assistants to inadvertently copy copyrighted code, leading to legal issues. To mitigate risks associated with AI-generated code, it's crucial to track its origin throughout the software supply chain. This requires distinguishing between human and AI-authored code. We first conducted a study investigating the feasibility of using lexical and syntactic features for this purpose. The results of our study were promising, it showed 92% accuracy and encouraged us to enhance our stylometric methods and delve deeper in the problem of detecting AI generated code. Next, we used a larger dataset with a bigger feature set for detecting AI-generated code. Our classifiers achieved up to 93% accuracy on standardized tasks indicating that it is possible to reliably differentiate between human and AI-generated code. Subsequently, we assess the resilience of these methods against adversarial attacks by using the LLM itself as an obfuscation tool. We introduce the concepts of LLM-based obfuscation and alteration attacks, demonstrating their efficacy in evading stylometric detection. The classifiers' performances were notably impacted by both obfuscation and alteration attacks. Recall scores dipped to 58% for obfuscation and 73% for alteration compared to the scores of our trained AI-code detection classifiers. This substantial decline in performances indicates the inability of the model to correctly identify AI-generated code when facing adversarial attacks. These results suggest that the attacks effectively disguised the AI-generated code, enabling it to bypass the classifier undetected. This underscores the challenges posed by these adversarial techniques and highlights the need for more robust detection methods.
dc.identifier.citation	Bukhari, S. A. (2024). Issues in detection of AI-generated source code (Master's thesis, University of Calgary, Calgary, Canada). Retrieved from https://prism.ucalgary.ca.
dc.identifier.uri	https://hdl.handle.net/1880/119794
dc.language.iso	en
dc.publisher.faculty	Graduate Studies
dc.publisher.institution	University of Calgary
dc.rights	University of Calgary graduate students retain copyright ownership and moral rights for their thesis. You may use this material in any way that is permitted by the Copyright Act or through licensing that has been assigned to the document. For uses that are not allowable under copyright legislation or licensing, you are required to seek permission.
dc.subject	Machine Learning
dc.subject	Large Language Models
dc.subject	LLMs
dc.subject	Code Stylometry
dc.subject	AI Code Assistants
dc.subject	ChatGPT
dc.subject	Software Supply Chain Security
dc.subject.classification	Engineering--Electronics and Electrical
dc.subject.classification	Education--Technology
dc.title	Issues in Detection of AI-Generated Source Code
dc.type	master thesis
thesis.degree.discipline	Engineering – Electrical & Computer
thesis.degree.grantor	University of Calgary
thesis.degree.name	Master of Science (MSc)
ucalgary.thesis.accesssetbystudent	I do not require a thesis withhold – my thesis will have open access and can be viewed and downloaded publicly as soon as possible.