Pigeon's Projects

Lots of the FOSS Galaxy users reported that they got emails about account lockouts. Looking at the GitLab logs, these were all from the same IP address. Someone was clearly trying to bruce-force GitLab logins. I’ve not seen this happen before, and it’s a slightly worrying turn of events.

Fast Resolution

First, GitLab omnibus stores data in a whole bunch of places. the log files are usually stored in ‘/var/log/gitlab/’, after looking around, I found the logs which showed the requests, including showing the failed login attempts (and what accounts were targeted):

cd /var/log/gitlab/gitlab-rails
nvim production_json.log
nvim production_json.log.1.gz

These showed the login attempts:

{
  "method":"POST","path":"/users/sign_in", ...,
  "params":[
    {"key":"user",
     "value":{"password":"[FILTERED]","login":"<user>"}},{"key":"authenticity_token","value":"[FILTERED]"},{...}],"remote_ip":"<ip address>","user_id":null,"username":null,"ua":"python-requests/2.9.1", ...}

I’ve abbreviated the logs for ease of reading, note [FILTERED] is something GitLab does itself – so passwords and keys don’t end up in logs :). The and are me to prevent these from appearing in the blog post.

What’s interesting is the user agent is python-requests, which implies that it’s a script (so almost certainly a compromised VPS or similar).

whois <ip address>

Doing a whois showed it was a small hosting company so I suspect that’s the case. When writing this blog post, more recent attempts are listed as “colocation at …” as the whois address field. So that’s probably the case.

As it was a single address, I blacklisted it using IP tables:

sudo iptables -I INPUT -s <ip address> -j DROP

This result in packets from this IP address getting dropped before being processed by GitLab.

Long Term Resolution: Fail2ban

Brute force attacks are nothing new, and there are tools out there to stop these kinds of attacks for a range of protocols (notably SSH). Gitlab does a reasonable job, by locking accounts after many attempts, but someone being locked out and getting an email to unlock their account is annoying, and results in a lot of emails when it’s every user!

We already use fail2ban to detect and (temporarily) ban SSH and email bruce force attacks. This has now been extended to Gitlab. I was prepared to write my own jail config, but it looks like someone beat me to it, so we’re just using that one.

The jail is now set up and it’s live on the server. I’ll monitor it over the next few weeks and see if it helps catch and ban these kinds of attacks. If it’s not doing a good enough job I’ll update with the modifications I made to fix it.

I’d also like better reporting of these kinds of attempts so we can keep a closer eye on it in future – but that’ll be part of the larger set of system monitoring upgrades I have planned.

When updating Gitlab this morning, there was a conflict caused by the update. This is on purpose because there is an issue with Gitlab 14.1.x –> 14.2.0 upgrade process which was fixed in the latest Gitlab 14.1 release (14.1.3).

First Attempt

The exact version needed wasn’t listed, so I needed to figure it out:

yum list --show-duplicates gitlab-ce

Will show all package versions in the repository. Then I found the latest 14.1.x release – just before the current release (14.2.0).

To force an update to a specific version, its package name is the name shown in the output of yum list with the version number tacked on the end:

yum update gitlab-ce-14.1.3-ce.0.el7

Edit: Even after the update, the upgrade still fails, waiting for a solution on this one :)

Transaction check error:

  file /opt/gitlab/embedded/service/gitlab-rails/db/ci_migrate from install of gitlab-ce-14.2.0-ce.0.el7.x86_64 conflicts with file from package gitlab-ce-14.1.3-ce.0.el7.x86_64

Issue Tracker Fix

Edit2: found a fix on the gitlab issue tracker:

mv /opt/gitlab/embedded/service/gitlab-rails/db/ci_migrate /opt/gitlab/embedded/service/gitlab-rails/db/ci_migrate_back
yum update

Usual caveats about messing with files managed by packages apply, but until an official fix is released it’ll do.

See also: https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6347 which mentions a merge to fix the issue.

Unfortunately, my employment with the University of Essex has come to an end. It's a little sad, as I very much enjoyed working with the amazing people in the department and all of the students I've had the pleasure to teach over my time at the University.

The good news is I've got employment at another University, as part of the team delivering their games degree modules. The university in question is Falmouth University. They're a lovely bunch and I'm very much looking forward to working with them. There was just one slight problem with the new job – it's the location!

I've lived in Essex my entire life. When I've rented for my undergrad degree it was either via the university or a dedicated student-housing landlord which handled pretty much everything for you. Trying to find somewhere in Cornwall from Essex proved challenging.

I had problems trying to find somewhere and arrange a booking during the week I'd managed to arrange as a trip to view properties. I can see why – the area is amazing. Thankfully, the very nice people at Real Estates managed to find me somewhere ( the apartment I'm in is very nice, and has everything I could have asked for ).

I've spent the weekend sorting out bills, trying to get recycling boxes, etc... and now that's all done I'm looking forward to getting back some of the work I need to do. At the moment I'm using a wifi hotspot provided by BT until the broadband is sorted. It 'sort-of' works but every now and again demands that I re-enter my credentials. This isn't something that's just me – the couple next door told me they had a similar problem when they first moved in.

Phone Tethering ('usb connection sharing')

The no-internet problem is something I'd tried to prepare in advance for. Firstly, the most sensible option would be using mobile-phone tethering. This is allowing the use of either the phone's WIFI connection or USB port to share the mobile connection from a phone to a computer. In the past being able to hook an android phone up and use it as a “USB ethernet adapter” has proven very useful on strange hardware without wireless drivers. You can also use this to turn the phone into a USB wireless dongle (connect to a wireless network, enable connection sharing over USB).

My mobile provider explicitly allows this, and it's been very useful in the past when internet issues have come calling or I've been away in hotels (read: travel lodges and premier ins). The problem with this approach is that I don't seem to get a phone signal inside the flat. I suspect the thick stone walls are to blame (but they keep it nice and cosey so can't complain too much :)).

The lack of phone signal does prevent another problem, however, namely that without the internet or a phone signal, setting up contracts for utilities is problematic. There is a signal outside, which is why I spent most of the weekend standing outside the flat talking to companies about setting up contracts.

With working internet, the phone signal problem can be mitigated thanks to a neat feature of modern phones, wifi calling. This allows you to make calls over the internet using your mobile number (similar to using SIP phones with trunking). This is all completely transparent when enabled.

In other words, if I have a phone signal I can have internet, or if I have internet I can have a phone signal. The problem is, I'm still without either.

4G modem

Plan B was similar to plan A, when looking at the flat originally it was noted that there was some signal with another UK mobile network, EE (my carrier being Three). In preparation for the move, I brought a 4G modem with a pay-as-you-go EE sim card. This approach works but is very unstable. There is an EE signal in the flat, but only barely. The maximum speed seen so far via this approach is about 200kbps, and even then it was patchy.

As a result, that plan was left on the drawing board.

BT WIFI

There are a few BT routers around, most of them have very good signal (I suspect from other occupants of the building). They are broadcasting an ID called BT wifi, which is a paid-for service that gives you access to the internet through any BT router participating in the scheme. You can also get free access to this if your router is part of the scheme, although I'm not sure how many people know that.

I decided to shell out for this as a work-around until the BT engineers can come and engineer some broadband for me (on a side note, my landlord has agreed to let me have fibre to the premises, which has made me absolutely ecstatic! I'll be able to get my work done with no problems!).

This solution isn't perfect, the connection drops somewhat periodically. I've not been able to find the cause of this but when it comes back it requires me to re-authenticate. I've tried disabling wifi address randomisation but had no luck with it. The connection seems to drop more when I'm moving around the flat (it's most stable if I sit at my desk) so I suspect it's switching SSIDs or something.

I'm hoping it'll be good enough for my Microsoft teams sessions at the start of term, else that the broadband is working before then!

I've not posted for the past two days as they've been fairly full-on marking. Usually, demos are done in person. The markers are in a Lab and the students come during their time slot and we go through the work with the student. Given the current situation, that's proved impossible.

Instead, we've been using zoom to do the marking. Zoom on Linux I've found to be a bit of a mixed bag. Firstly, when using GNOME it was fairly easy-going. I've not used zoom on Windows but I suspect it would have been a fairly similar experience. I've not tried it on Wayland with Gnome (or if I have, I've not noticed), but I suspect it'll be similar to the point below.

The problems I've had started when I tried to use Zoom in Sway. Firstly, it works reasonably well the application can launch, and can do all the basics it can in gnome. This shouldn't be too surprising as it's mostly the same technology stack (pulse audio, whatever they're using for video input (v4l?), etc...). Screen shares from other users also work OK.

However, there are three things that have been fairly annoying when trying to use it.

Meeting Scheduling workflow

Firstly, Zoom's integration into calendars seems to be somewhat lacking. There are two or three important features when dealing with meetings. Firstly, I'd like to know the topic. Zoom's got us covered there, big title box. Second the date and time would be nice. There is a calendar for entering the date, it's stuck in American date format (m/d/y) which is annoying if you forget and your meeting's now in December. The time entry field is the most bazzar on sway you get a single integer for the hour and a single integer for the minute. Sometimes, they're visible most of the time they are not. There were similar issued in gnome, but it was at least usable.

I suspect if I had outlook installed locally on a Windows machine, it'd be passable, but using the office365 calendar with it is a pain. Firstly, you need to create the meeting in Zoom, to get an ID, which then generates a meeting ID. So far so Good, It also generates a calendar invite, but this isn't integrated into office365 and you can't seem to invite participants via zoom (despite the fact that it's integrated into zoom on the server-side for logins and the invite panel once the meeting has started).

Instead, what you need to do is either: copy the invite text into a meeting in the web interface, re-enter the meeting details and then invite your participants via outlook; or you can download the calendar invite locally, create an email, add the participants to the email and then attach the meeting request. Once sent you can then add it to your office365 calendar using the “add to calendar” from your sent items. It doesn't seem there is a way to add the event directly from the calendar screen (unless there is and i missed it). Either way, the other participants then get your meeting request. If you need to alter the meeting for some reason (say, for example, you forget it's bank holiday on Friday) then you need to edit it in both Zoom and Office365. When you have a large number of meetings, this can get very tedious.

For the demos the work-flow was slightly different, create the meeting. Get everything set-up (code up and running, marking sheets open, etc...) then invite the student. I did this via email, because the 'invite via zoom' feature doesn't seem to notify them if the client isn't open (or at least, that's what the students have said).

The way to do this is to open up the participants panel, click the invite button and then click on the copy to clipboard button. I'm not sure who named the button, but I suspect it's shorthand, because, “Copy to clipboard, except sometimes not the clipboard you where expecting but somehow still shows up if you paste in vim, just not in Firefox” probably was a bit wordy. I found myself pressing the button, opening up vim, pasting the contents so I could re-copy it in vim and then paste it in Firefox. Pasting into vim was also was very slow (it would take a few seconds for the text to appear). This doesn't happen with any other application, just Zoom. I have no idea why this was happening, but I really wish it wouldn't.

But, now you've scheduled your meeting, and it's meeting time.

Magical Disappearing windows

Secondly, (and I remember this happening in a webinar in GNOME as well) when you start screen sharing or stop sometimes the pop-out windows (chat, participants, Q&A) appear and disappear at will. I suspect this might be a feature rather than a bug. If you only have one screen maybe it's useful to hide all of the zoom nonsense when screen sharing starts. Every time you start or stop screen-sharing this will happen again.

It slows down the flow somewhat if you need to stop giving your material, to find out on what intergalactic plane your student's chat window has decided to migrate itself to this time. The same thing also happened to the main window when a meeting started. For the demos, where you want to keep an eye on the marking chat (so you know if one of the other markers needs assistance or has an urgent query) this is quite disruptive. In sway this was much better. Windows seemed to disappear less often, there are a few graphical issues (half of the user's face) in the meeting window itself, but float it and it's fine (and I'm used to this happening on applications that aren't designed for tiling window managers).

I have no idea of the steps involved in making it happen, but sometimes zoom would open up into a single window mode where the chat, participants, etc.. were on the side, rather than creating lots of small windows. This was much nicer especially when floating, Zoom, can I have that as default? :D

No, oh, okay then.

Screen sharing

This last one isn't Zoom's fault. Screen sharing doesn't work in sway. I'm not surprised, it's Wayland based wlroots has it's own screen sharing API and I wouldn't expect they had considered it for for the 4 or so Sway users. I have ways of working around this one mostly. What would be quite nice is if I could RTMP-stream to a webinar rather than having to use Zoom's share desktop feature.

When using Sway, I cannot give a lecture. I can't show code on my screen or go through an exam paper if I can't show the exam paper or code editor! My work around, or at least my first work-around before the lockdown proper, was to use OBS and the plugin I mentioned in my last post. The lecture then became a youtube livestream. After the lockdown proper, when I was giving my lectures from home, I used gnome shell and put off reinstalling my OS until after my last revision lecture (and Fedora 32 was out). By happy co-incidence that happened to be the same week.

Ok, this post was quite negative, lets end on a good note.

Overall, I'm actually happy they provide a Linux client at all. It's a bit rough around the edges, and of course I'd prefer something a little more polished, but it's better than nothing. All the basics work quite well and I've not found anything insurmountable. Of course, I'd prefer it was FOSS, but it's the 'standard' that's been settled on for the time being. The sway-specific issues are somewhat self-inflicted, I use sway because it makes my life easier, but I don't expect them to support it. So, thanks for letting me (mostly) get work done from home Zoom, just hope I don't actually need to still be doing this in real December.

#100DaysToOffload

Today has been a fairly lazy day. So not much to blog about, and not much comes to mind to write about.

I've spent a little time with vscodium and Flatpak today. I'm working on porting one of my Java projects to C++.

Trying to get VSCodium to stop yelling about includes was providing difficult. I do like using Meson. When the code is more than a single class I'll put it on gitlab :).

Oh, and the bug I reported yesterday got fixed \o/

#100DaysToOffload

Fedora 32 was released during the week, I've been putting off sorting out my mess of a desktop for a few months and this seemed like a good opportunity.

Before I started the upgrade, I made two Fedora 32 USBs. One of them had the standard live image on, and the other had the 'everything' install. The reason I created both was a little silly – I wanted to use the live Image to back up the data to my NAS before formatting and then was planning on using the 'everything' USB stick to install (as I could select only the packages I wanted).

Of course, I could have used the liveCD to install. I've been using Sway on my work machine since we moved office (blame Dan), but I noticed I had problems with default applications when using Sway. As a result, I re-installed but only kept the applications that I planned on using (not having any of the GNOME packages I didn't need).

Gnome shell on the Fedora 32 liveCD looks really nice. You can see little UI changes all over the place where the devs have been working hard to improve it. Still, I've gotten used to my workflow in the office and working from home I want to try and mimic it as much as possible.

First problem was that I couldn't figure out how to create a LVM group the way I wanted from the Fedora Installer (two disks, one with /, /boot, /boot/efi and one with /home). I gave up in the end and symlinked stuff from the second disk into /home.

It's taken me most of the day, but I've got my home machine back into a usable state, with all it's theming and tools.

#100DaysToOffload

Oh, and I even reported a possible bug against the OBS studio plug-in I use for Sway.

Today was the last of the three revision lectures that I was giving before the exams start.

The revision lectures are usually in the lecture halls, but with everything that's going on we are doing them from home. It's a bit of a mixed bag. Important that we stay inside, however, I miss being on campus.

It's not quite the same being in a room by yourself, speaking into the void as it is speaking to a room full of students. There isn't anywhere near the same level of visual feedback during the lecture.

I also miss being able to walk around the labs and see all the awesome stuff students are working on. Hopefully, we can all come back next term.

#100DaysToOffload

Blogging on the fediverse, an interesting idea and a possibly simplified way of doing this.

It's been a while since I regularly blogged, and if I'm being honest it wasn't all that regular. I've been looking at the 100 days of blog post idea that's been on the fediverse the past few days and thought I'd jump in with both feet – not as if there is a lot else to really do at the moment (besides work that is :)).

Anyways, I'll post something proper tomorrow. It's midnight so today's post is simply a hi, I'm about, nice to be here post.

#100DaysToOffload