Some time ago I stumbled upon a website called Sad Servers, which challenges users to fix broken Linux machines.

It seemed interesting since I'm actually a baby in Linux-land. I only dedided to change my notebook's operating system from annoying Windows to Pop_os less than two years ago after being content with WSL (Windows Subsystem Linux). I just couldn't call myself a computer enthusiast without trying it.

Anyways, I forgot about it, then I got lazy. But I decided I keep avoiding a lot of suff I really want to do over nothing, so I just grabbed the website one day and started playing with it.

I only got through 4 scenarios from the Easy List, but each one taught me something new about troubleshooting. Down below you can see my implementation, which I don't claim to be the best approach, but I think that's the whole point.

Scenario 1 - Saint John

Scenario: "Saint John": what is writing to this log file? Level: Easy Description: A developer created a testing program that is continuously writing to a log file /var/log/bad.log and filling up disk. You can check for example with tail -f /var/log/bad.log.
This program is no longer needed. Find it and terminate it. Do not delete the log file.

Test: The log file size doesn't change (within a time interval bigger than the rate of change of the log file).
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Root (sudo) Access: Yes

My Solution:

This server is very simple, really exists to explain how the scenarios work. The first thing I did was to run the actual command the description told me to: tail -f /var/log/bad.log. It's a file that continues to print numbers.

Ok, the next thing to do is finding the PID of this process. By finding it, we're able to manage it.

lsof /var/log/bad.log will return to me the PID of the process running it and now all we need to do is kill it. Running kill PID will fix the issue.

For example, in my case the lsof command returned a PID 542, so I should write kill 542 and that will kill the process with this PID.

Scenario 2 - Saskatoon

Scenario: "Saskatoon": counting IPs. Level: Easy Description: There's a web server access log file at /home/admin/access.log. The file consists of one line per HTTP request, with the requester's IP address at the beginning of each line.

Find what's the IP address that has the most requests in this file (there's no tie; the IP is unique). Write the solution into a file /home/admin/highestip.txt. For example, if your solution is "1.2.3.4", you can do echo "1.2.3.4" > /home/admin/highestip.txt

Root (sudo) Access: False

Test: The SHA1 checksum of the IP address sha1sum /home/admin/highestip.txt is 6ef426c40652babc0d081d438b9f353709008e9 (just a way to verify the solution without giving it away.)

My Solution:

My first instinct was running this: uniq -w 15 -c /home/admin/access.log | sort -h but obviously it didn't work, since I was grabbing an arbitrary number of characters and then sorting them as human readable numbers.

Hm, I actually don't know a lot of commands since I'm a baby in Linux world, but the whole point of going through these challenges is to grow the size of my baby brain.

Ok, I made some assumptions, but let's analyze the file correctly. This is an example from a line of the file:

180.76.6.56 - - [20/May/2015:21:05:56 +0000] "GET /robots.txt HTTP/1.1" 200 - "-" "Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2"

So I need to find a way of testing only the IP portion of it against each other. First thing I need was changing the sort flag from -h to -n and adding an r as well for: reversing the order. Which means the high will be at the end, simplyfing my life. The second thing was trying to grab only the portion of the line like this:

uniq -c -w 12 /home/admin/access.log | sort -nr

It also didn't work, simply because some IPs were using IPv6 structure, so they weren't counted.

I ended up with this, after searching around for what awk actually does. I find its syntax overly complex and weird, but apparently I can use it to print only one portion of the line like this:

awk '{print $1}' /home/admin/access.log | uniq -c | sort -nk 1

Tiny note: I kept writing aws all of the time because I'm much more inclining of using their package at work.

I also changed the sort to use the -k flag because I can set it to sort via a key, which in this case, it's the printed word piped from awk.

But I still got the wrong answer.

Now I was already fairly confused, so I printed out the whole thing and tried analyzing it manually.

While looking at it, I noticed this:

...
2 66.249.73.135
2 66.249.73.135
2 66.249.73.135
2 66.249.73.135
2 66.249.73.135
2 66.249.73.185
2 66.249.73.185
2 66.249.73.185
2 66.249.73.185
2 66.249.73.185
2 66.249.73.185
2 66.249.81.20
2 66.249.81.20
2 66.249.81.91
2 66.249.83.223
2 66.249.83.239
...

Why aren't these lines being treated as equal? Something was weird in my command.

Again, I tried searching online and found someone in StackOverflow saying that it could be differences in whitespace or something but they suggested using either RegEx or awk '{ print $1 }' to remove it. Well... what?

Then I looked at the man pages for uniq again.

I tried adding a very funny flag to it, which is -u. I find it funny because it shows unique lines, which the command should already do, right?

awk '{print $1}' /home/admin/access.log | uniq -u -c | sort -nk 1

But that did not work like I expected, it printed 1 line for each instance that the IP appeared, so I just got all the duplicates. Huh.

But I found yet something else. I accidentally lost the link so if anyone find the discussion for this I'd be very happy!

The discussion was that we should run sort before the uniq. So I tried actually adding one extra sort to it because I still wanted to sort later to grab the result. So this one is the final command:

awk '{print $1}' /home/admin/access.log | sort | uniq -c | sort -nk 1

And I actually got the right answer this time.

I have no idea why we need to use sort before uniq, but I'll keep that in mind. Baby brain has grown.

Scenario 3 - "The Command Line Murders"

Not gonna work through this one because the whole point is to conduct the investigation. I'm not kidding, it's gonna ruin it if I explain it here, but I can attest that my approach was reading the files and using only the basic shell tools, you'll cd, cat and grep a lot for this one.

Scenario 4 - "Taipei": Come a-knocking

Level: Easy Description: There is a web server on port :80 protected with Port Knocking. Find the one "knock" needed (sending a SYN to a single port, not a sequence) so you can curl localhost.

Root (sudo) Access: False Test: Executing curl localhost returns a message with md5sum fe474f8e1c29e9f412ed3b726369ab65. (Note: the resulting md5sum includes the new line terminator: echo $(curl localhost))

My Solution:

I have no idea where to start with this.

I searched around for what port knocking is and found an answer that I could somewhat understand. Basically, it's a technique that treats your ports as combinations for a lock in order to strengthen security. If you have a port :33 you want to secure, it can only open after we send some info to ports :80, :2222 and so on. And you need to send your packets in order, otherwise you'd need to restart.

I don't think it's widely used but still interesting to find.

I had to use all the clues to get this one working, but anyways, the exercise comes with this lib called knock which we can use to check on some ports. You can read about it here.

I can also use nc (netcat) but simply by using it, it got me nowhere since I don't know which ports are available.

Searching for "how to know how many ports are available in a sever" got me nowhere actually, which makes sense. Actually, it did, but running netsat -a got me nowhere.

There's also another clue indicating a method, which consisted in writing a bash loop going through 1 to a very high number and knocking on each port. But that sounded like it was going to get me locked out from it (later I discovered that I actually wasn't, the exercise was just to knock on the ports).

So I tried the final clue: nmap -p- localhost on it. Well, it turns out that it did give me the connections I wanted. So I made a list from them and created the bash script to hit on them until I got the response:

sq=("22" "6767" "8080")
for i in $sq; do
  knock localhost $i
  curl localhost 2>/dev/null && echo "my port - $i" && break
done

The response is one of the three I showed (I also don't know if the server always gets new ones so this might be useless to you). And then I just did what the description asked me to: echo $(curl localhost)

This one was interesting from the start and actually expanded a bit of my knowledge so that was nice.

Conclusion for now

Those take me more time than I expected, probably because of my lack of experience with such scenarios. I'll keep practicing and learn more about sysadmin before trying another batch. Hopefully, that will come soon!

Wanna discuss this article with me? You can send me a comment in the neocities profile with this post code as reference: 002