At some point, I discovered something you never really want to find on a server you manage: a phishing-style sign-in page sitting inside one of my websites.

My first thought was the obvious one:

How far did they actually get?

Finding a malicious file is uncomfortable, but it also creates a dangerous temptation to jump straight to the worst-case scenario. Root compromise. Stolen databases. Every site on the server infected.

What I learned while investigating this incident is that the presence of a phishing page proves something much narrower:

Someone found a way to write a file somewhere they shouldn't have been able to.

That is serious. But it does not automatically mean they owned the entire server.

The useful part is figuring out what happened next.

My first priority wasn't deleting the file

The instinct when you find something malicious is to delete it immediately.

I didn't want to do that.

Deleting the file might remove the visible problem, but it can also destroy one of the most useful pieces of evidence you have.

So I first isolated the affected area and preserved a copy of the suspicious file somewhere visitors couldn't reach it.

Something along these lines:

SITE_ROOT="/srv/www/example-site"
CASE_DIR="/root/ir-case"
SUSPECT="$SITE_ROOT/path/to/suspect-file"

sudo install -d -m 0700 -- "$CASE_DIR"
sudo cp -a -- "$SUSPECT" "$CASE_DIR/"
sudo sha256sum -- "$CASE_DIR/$(basename "$SUSPECT")"
file -- "$CASE_DIR/$(basename "$SUSPECT")"

Those paths are deliberately generic here. I'm not publishing production paths, usernames, IP addresses, or raw logs from the actual incident.

The important idea is simple:

Contain first. Preserve second. Investigate third.

I also avoided executing the file or casually opening it in a browser.

One thing I had to keep reminding myself of was that timestamps are clues, not absolute truth. A copied or manipulated file can carry timestamps that make the timeline look different from what actually happened.

So instead of trusting the suspicious file alone, I started looking for independent evidence.

The logs started telling the story

This was where the investigation became much more interesting.

I compared authentication activity, file-transfer activity, and web requests around the period when the suspicious file appeared.

On a systemd Linux server, SSH activity can be inspected with something like:

sudo journalctl --since "24 hours ago" -u sshd --no-pager
sudo last -ai | head -50

Depending on the distribution, the service may be called ssh instead of sshd.

I then looked at the relevant transfer-service logs and compared those timestamps with requests in the site's access logs.

For example:

WEB_LOG="/var/log/example-web/access.log"

sudo grep -hE '"(GET|POST|PUT) ' "$WEB_LOG"

I wasn't interested in dumping thousands of lines onto the screen.

I wanted three things:

  1. When did the first suspicious change happen?

  2. Was there a successful login or file transfer shortly before it?

  3. What happened after the phishing page appeared?

That distinction matters.

A malicious file tells you what exists.

Logs can help tell you how it got there.

And those are two very different questions.

Finding one bad file wasn't enough

Once I had a rough timeline, I needed to know whether the phishing page was the whole incident or merely the most visible part of it.

So I checked recently modified files:

sudo find "$SITE_ROOT" -type f -mtime -2 \
  -printf '%TY-%Tm-%Td %TH:%TM %p\n' | sort

I also searched script files for some common suspicious PHP patterns:

sudo grep -RInE \
  'eval[[:space:]]*\(|base64_decode[[:space:]]*\(' \
  "$SITE_ROOT" \
  --include='*.php'

This is useful for triage, but there's an important catch.

eval() and base64_decode() do not automatically mean malware. Legitimate software can use them, and competent attackers can easily avoid obvious patterns entirely.

So I treated matches as clues rather than verdicts.

That mindset turned out to be useful throughout the investigation: security tools give you evidence, not magical yes-or-no answers.

The most important question became permissions

This was probably the biggest lesson I took away from the incident.

Once I knew an account or publishing mechanism could write into the site, I stopped thinking only about the malicious file.

I started asking:

What else could the same account reach?

Could it read application configuration?

Could it see database credentials?

Could it modify another site?

Could it access SSH keys?

Could it create scheduled tasks?

Could it reach mail credentials or API tokens?

I checked timers and running processes as part of that review:

sudo systemctl list-timers --all
sudo ps auxf

I also reviewed application administrators, SSH keys, database access, scheduled jobs, permissions, and anything else the affected account could potentially touch.

This is where a lot of incident investigations can go wrong.

Finding and deleting phishing.php feels satisfying because you've removed something tangible.

But the actual security boundary is usually the account and its permissions, not the individual file.

If that account could read five other secrets, those five secrets have to be considered part of the incident too.

I rotated credentials based on reach, not panic

Another thing I learned was that “change every password immediately” isn't quite specific enough.

Credentials need to be rotated according to what was actually exposed.

The compromised publishing credential came first.

But even that required understanding what kind of account it actually was.

A local Linux account, an FTP virtual user, and an account managed by a hosting panel are not necessarily changed in the same place. Running passwd on Linux doesn't magically reset every FTP credential configured elsewhere.

After that, I rotated secrets the affected account could potentially read:

  • application administrator credentials

  • database passwords

  • mail credentials

  • API tokens

  • reused passwords

  • active sessions and authentication tokens where possible

This also made me much more conscious of credential isolation.

If two unrelated services share the same password or an account has access to things it never needed, one small compromise suddenly gets a much larger blast radius.

The real fix was making another upload less dangerous

Cleaning the incident was only half the job.

I didn't want to merely remove the phishing page and hope it never happened again.

I wanted the same attack to be much less useful next time.

For uploaded or static-content directories, one of the strongest protections is simple:

They should not be allowed to execute scripts unless there is a very good reason for them to.

That rule should ideally live in configuration controlled by the server administrator, not somewhere the compromised publishing account itself can rewrite.

Before reloading a web server after changing configuration, I also validate it:

nginx -t

or:

apachectl configtest

I tightened the publishing account as well.

If an account only exists to upload files to one website, it probably doesn't need access to another site's directory, a shell, the database, mail, cron jobs, and half the server.

That sounds obvious when written down.

It becomes much more obvious after you've had to investigate what a stolen credential could actually reach.

SFTP doesn't solve a stolen password

This was another distinction worth keeping.

Moving from an unencrypted transfer protocol to SFTP or FTPS is absolutely useful because it protects credentials and traffic in transit.

But encryption doesn't protect you from someone who already has valid credentials.

If an attacker has the correct username and password, SFTP will very securely encrypt their connection too.

So transport security and account security solve different problems.

Where practical, I now prefer combining things like:

  • unique credentials

  • MFA

  • limited permissions

  • isolated site directories

  • non-executable upload locations

  • IP restrictions or VPN access for management

  • monitoring for unexpected file changes

No single one of these is magical. Together, they reduce how much damage one stolen credential can do.

A clean malware scan doesn't prove nothing happened

One of the more dangerous conclusions during an incident is:

“The scanner found nothing else, so we're clean.”

A scanner can tell you what it found.

It cannot prove what it didn't see.

The same applies to logs. An empty log from one service doesn't prove no access occurred if the relevant activity happened somewhere else, logging was disabled, or the data had already rotated out.

That's why I ended up thinking about the incident in terms of scope, rather than simply “infected/not infected.”

What could the compromised account access?

What evidence showed that access was actually used?

What couldn't I determine from the available evidence?

Those questions were much more useful than searching endlessly for a definitive green check mark.

Recovery came after understanding the incident

Once I understood the likely path and scope, I could recover much more confidently.

I restored affected files from a known-good source, patched what needed patching, tightened permissions, rotated exposed credentials, and continued monitoring authentication, web, transfer, file-change, and outbound activity afterward.

I also made sure backups weren't writable through the same publishing account.

Because a backup that an attacker can modify alongside the live site isn't much of a backup.

And yes, testing whether you can actually restore those backups matters too. “We have backups” and “we can restore this server from them” are surprisingly different statements.

The three questions I would ask first today

If I discovered the same kind of phishing page again, I wouldn't begin by asking:

“Have they hacked my entire server?”

I'd start with three much more useful questions:

What was uploaded?

What could the account that uploaded it reach?

What evidence shows that the attacker went beyond that?

That change in thinking made the incident much easier to reason about.

The phishing page was the visible symptom.

The real investigation was about access, permissions, and evidence.

And the most important fix wasn't simply deleting a malicious file.

It was making sure that the next stolen credential - if there ever is one - has far less power than the last one did.