Pigeon's Projects

Lots of the FOSS Galaxy users reported that they got emails about account lockouts. Looking at the GitLab logs, these were all from the same IP address. Someone was clearly trying to bruce-force GitLab logins. I’ve not seen this happen before, and it’s a slightly worrying turn of events.

Fast Resolution

First, GitLab omnibus stores data in a whole bunch of places. the log files are usually stored in ‘/var/log/gitlab/’, after looking around, I found the logs which showed the requests, including showing the failed login attempts (and what accounts were targeted):

cd /var/log/gitlab/gitlab-rails
nvim production_json.log
nvim production_json.log.1.gz

These showed the login attempts:

{
  "method":"POST","path":"/users/sign_in", ...,
  "params":[
    {"key":"user",
     "value":{"password":"[FILTERED]","login":"<user>"}},{"key":"authenticity_token","value":"[FILTERED]"},{...}],"remote_ip":"<ip address>","user_id":null,"username":null,"ua":"python-requests/2.9.1", ...}

I’ve abbreviated the logs for ease of reading, note [FILTERED] is something GitLab does itself – so passwords and keys don’t end up in logs :). The and are me to prevent these from appearing in the blog post.

What’s interesting is the user agent is python-requests, which implies that it’s a script (so almost certainly a compromised VPS or similar).

whois <ip address>

Doing a whois showed it was a small hosting company so I suspect that’s the case. When writing this blog post, more recent attempts are listed as “colocation at …” as the whois address field. So that’s probably the case.

As it was a single address, I blacklisted it using IP tables:

sudo iptables -I INPUT -s <ip address> -j DROP

This result in packets from this IP address getting dropped before being processed by GitLab.

Long Term Resolution: Fail2ban

Brute force attacks are nothing new, and there are tools out there to stop these kinds of attacks for a range of protocols (notably SSH). Gitlab does a reasonable job, by locking accounts after many attempts, but someone being locked out and getting an email to unlock their account is annoying, and results in a lot of emails when it’s every user!

We already use fail2ban to detect and (temporarily) ban SSH and email bruce force attacks. This has now been extended to Gitlab. I was prepared to write my own jail config, but it looks like someone beat me to it, so we’re just using that one.

The jail is now set up and it’s live on the server. I’ll monitor it over the next few weeks and see if it helps catch and ban these kinds of attacks. If it’s not doing a good enough job I’ll update with the modifications I made to fix it.

I’d also like better reporting of these kinds of attempts so we can keep a closer eye on it in future – but that’ll be part of the larger set of system monitoring upgrades I have planned.