How can an AI "good student" become a "hacker"? The abyss of technological ethics is cracking

The Tokyo Metropolitan Police Department of Japan recently arrested a 17-year-old high school student from Osaka. He was accused of using ChatGPT to generate attack processes and malicious codes to carry out cyber attacks on the Internet cafe chain "Kuaihuo CLUB" and its related businesses. A total of 7.24 million requests were initiated, which may have led to the exposure of data of approximately 7.3 million customers. This "genius teenager" who won an award in a cybersecurity competition committed the crime simply because "it was interesting to discover the vulnerability". The incident quickly sparked discussions about the deep-seated risks brought about by the popularization of generative AI: as technological democratization sweeps through society, the threshold for crime is also being rapidly lowered.

This case has demonstrated a new path of "question-and-answer crime". In the past, cyber attacks required years of technical accumulation. But now, criminals only need to have proficient conversations with AI to obtain the principles of vulnerabilities, attack methods, and even automatically generated code drafts. AI has transformed the manual skills of traditional hackers into process-oriented and standardized "assembly lines". Security agencies have discovered that malicious software is embedded with large language models, enabling it to generate code that evades detection in real time, leaving traditional anti-virus systems that rely on feature recognition at a loss.

A more covert approach is "indirect prompt injection" - attackers tamper with web pages or public text, causing AI assistants accessing these contents to mistakenly execute hidden instructions, thereby stealing user sessions or enterprise information. The attack entry has permeated into daily usage scenarios, making the defense boundary extremely blurred.

However, the lowering of the technical threshold is only a superficial risk. The deeper threat lies in the impact on social trust and the legal system. First of all, the portrait of the criminal subject becomes blurred. A teenager who acts impulsively out of curiosity and "for fun" can create data incidents comparable to those at the national level. Japanese data shows that nearly 70% of the suspects in illegal access cases are young people. The empowerment of technology leaves almost no buffer space between pranks and serious crimes.

Secondly, AI has given rise to new types of crimes that are difficult to define and hold accountable. In Japan, there have been cases where teachers were prosecuted for possessing AI-generated deepfake child pornography images. For the first time, the judiciary has determined that "completely virtual" illegal images are also illegal. This has shaken the traditional legal framework based on "real victims", creating a sharp conflict between creative freedom and the protection of minors.

Thirdly, risks have seeped into the technical underlayers. Research shows that by jailbreaking or exploiting memory function vulnerabilities, persistent malicious instructions can be injected into ChatGPT to make it execute the attacker's intentions for a long time. We rely on intelligent assistants that enhance efficiency, but we might be "turned upside down" without the users' awareness.

In this ironic picture, advanced technology, which was originally intended to improve life, can easily become a tool to disrupt order. To address this era's challenge, it is necessary to build a systematic solution that spans governance, technology and education.

At the governance level, the law needs to strive to stay ahead of technology. Japan is exploring the inclusion of deepfakes within the scope of severe punishment. China has launched special campaigns such as "Clear and Bright" to crack down on the abuse of AI and has promoted the implementation of the "Measures for the Identification of AI-Generated and Synthesized Content", mandatoring the identification of AI-generated content and providing a model of "agile governance" for the world.

At the technical level, defense must enter a stage of "AI against AI". Enterprises should not only utilize AI but also build an "immune system" for the AI era - such as implementing sandbox isolation for model access rights, continuously conducting adversarial tests, and proactively identifying potential vulnerabilities.

At the social and educational levels, when AI becomes a toolbox for all, digital ethics and legal education must become basic general knowledge. What society needs to cultivate are not only people who can use tools, but also digital citizens who understand the double-edged nature of technology, maintain a sense of awe towards risks, and have legal awareness.

The Japanese high school student case serves as a global alarm bell, heralding the advent of an era of "attacking democratization". AI itself has no morality; it is merely an amplifier of power, amplifying both creativity and the desire to destroy. The future security defense line will not only rely on programs and firewalls, but also on whether society can form a stable consensus on risks in terms of law, ethics and education. If consciousness and systems fail to keep up with the rapid pace of technological advancement, the price society pays for this may far exceed the benefits it brings.