2021-02-28 20:44:00
Let me tell you! When you're 11.5 hours into a 12 hour exam, this is NOT a screen you want to see on your main Gitlab that holds all your exam code. ( O_o)
Thank ${Deity} I cloned it all to my local system.
To clarify that a little bit: the CDP exam I took today is a practical exam where you spend twelve hours hacking, testing and building code that manages an application infrastructure. The whole exam, like the labs during class, are "in the cloud" run by Practical DevSecOps.
Around 1700, while trying to deploy a Docker container or two, my Gitlab runner became unresponsive and my Docker daemon died. Then the app webserver died. And then other students started piping up in chat that their labs were stuck.
Finally, around 1730 my Gitlab server (which holds all my exam code) was reprovisioned. That is: erased, rebuilt, re-installed. My work for the past eleven hours was gone.
So as I said: thank ${Deity} I had cloned my git repositores to my local machine.
View or add comments (curr. 0)
2021-02-26 10:55:00
Just some quick notes I've been making on how to quickly get gitlab-runner up on a Linux box. I still feel very yucky about curl-in a file into sudo bash, so I'll probs grab the file locally instead and make sure it doesn't do anything nasty.
The following example was used on my Ansible host, to install gitlab-runner and to have it run as the local "ansible" user account instead of root. It registers and starts two runners.
kilala.nl tags: work, studies,
View or add comments (curr. 0)
2021-02-20 17:13:00
Heh, it's a bit ironic, no? Six weeks ago I wondered whether I was over-doing it, with work and my studies. I'd just finished a few course and two exams and was about to start with a new client.
Not two weeks later I've taken another two classes and I'm about to take another exam. A twelve hour, practical exam followed by documentation and reporting.
I've promised myself that, once I'm done with the exam, I'll spend a few weeks on nothing but gaming! Genshin Impact here I come! :)
EDIT:
Ah. I just realized: I start teaching class again in 6-8 weeks. That'll require prep-time too :D
View or add comments (curr. 0)
2021-02-20 16:10:00
After finishing the awesome BHIS "Modern Webapp Pen-testing" class (January), I immediately rolled into the "Certified DevSecOps Professional" course. I am lacking in experience with CI/CD, while having to support DevOps engineers every day.
The CDP labs by Practical SecDevOps are okay, but only testing Django.NV got stale.
What better way to learn about SAST, DAST, SCA and more than by running our beloved Juice Shop webapp through my own CI/CD pipeline?! :D
Not only does this give me a private Juice Shop in a safe environment (my homelab), but it got me more familiar with Gitlab and all the things that come with DevSecOps / SecDevOps / Security in DevOps / however you wanna call it.
The image above shows the Juice Shop project in my Gitlab, with its security testing and deployment stages. The last "Compliance" stage (with Inspec) didn't fit into the pic.
Running the pipeline builds a Docker image for Juice Shop, runs SAST, SCA, secret scanning and linters, then runs the Docker image on my testbox and runs Nikto, ZAP and SSLyze against it as DAST. All very much default/basic, but it's a start!
kilala.nl tags: work, studies,
View or add comments (curr. 0)
2021-02-19 07:42:00
I've been mentioning Gitlab for a while now and you might wonder why the sudden change. :D I'm working my way through the CDP training from Practical DevSecOps.
I needed a crash course that took me through a practical example of CI/CD pipelines, from A to Z, in a hurry. I'm in security and I need to advise DevOps engineers who work on those pipelines every day. I found it harder and harder to relate to them without having gone through their journey myself. Intellectually I understood most of the concepts, but everything stayed very vauge without me actually doing it hands-on.
So far the course is a resounding "okay". It's not wonderful, it's not bad, it's just that: pretty good. The slide decks are decent, the trainer narrating the videos has a nice voice, but the narration is quite literally reading from the text book. Some of the text on slides and in the labs was lifted directly from third party sources such as projects' Github pages or from articles like Annie Hedgpeth's series on running Inspec.
They have a huge amount of online labs, which is good, even if they get repetitive. So what I've done is setup Gitlab in my homelab as well, and apply all the things the course teaches me to multiple intentionally-vulnerable web apps.
So I've got Git repos for Juice Shop (Node.JS and Angular), django.nv (Python and JS), Webgoat (Java), GoVWA (Go) and others, which I'm treating like they were projects for my simulated company. Each of these gets its own CI/CD pipeline to run code quality checks, SAST, DAST and automated build + deploy through Docker.
It's been one heck of a learning experience and I'm looking forward to the closing exam, which is another 24h practical exam. I love those!
kilala.nl tags: work, studies,
View or add comments (curr. 0)
2021-02-17 20:30:00
When using a "shell" executor with gitlab-runner you may run into the following errors, when trying to upload artifacts to Gitlab.
ERROR: Uploading artifacts as "archive" to coordinator... error error=couldn't execute POST against https://gitlab.corp.broehaha.nl/api/v4/jobs/847/artifacts?artifact_format=zip&artifact_type=archive: Post https://gitlab.corp.broehaha.nl/api/v4/jobs/847/artifacts?artifact_format=zip&artifact_type=archive: proxyconnect tcp: tls: first record does not look like a TLS handshake
The issue here is that your "gitlab-runner" user account has picked up a http proxy configuration that's not sitting well with it.
In my homelab, the proxy settings are configured for all users using Ansible, through "/etc/profile". For the "gitlab-runner" user that apparently may be problematic when trying to talk to the internal Gitlab server. Quick and dirty work-around: unset the proxy settings from your environment.
echo "unset http_proxy; unset https_proxy" >> ~/.bashrc echo "unset http_proxy; unset https_proxy" >> ~/.profile
kilala.nl tags: work, sysadmin,
View or add comments (curr. 0)
2021-02-17 19:35:00
As part of the CDP course we're running unattended ZAP scans as part of integration testing, using the "owasp/zap2docker-stable" Docker container. The course materials tell you to run the CI/CD task using "docker:dind", a Docker-in-Docker solution. For some reason my Docker boxen aren't a fan of that; I'll have to debug that later.
Trying to run the ZAP container with a simple "shell" executor through gitlab-runner led to some fun challenges though! The course material suggests the following Docker run command:
docker run --user $(id -u):$(id -g) -w /zap -v $(pwd):/zap/wrk:rw --rm owasp/zap2docker-stable zap-baseline.py -t https://target:port -J zap-output.json
To sum it up: start the ZAP container, run the ZAP baseline script using your current UID and GID, mount your local directory as /zap/wrk and then write the results as a JSON file onto the mounted local directory.
This approach fails in two ways if you're not doing the fastest, dirty approach: running as the "root" user account.
Either you use it with "--user $(id -u):$(id -g)" and then you get the error message "Failed to start ZAP :(". Or you run it without that setting, then ZAP runs but it cannot save the output file, with a "permission denied: /zap/wrk/zap-output.json" message.
The issue here is that container has a very limited setup of users (as it should) and your uid+gid are most likely not in there. Under normal conditions, the ZAP scripts inside the container run as "zap:1000:1000" but that user doesn't have write access to your user's directory on the Docker host.
So... If you're running the ZAP container directly on your host and not as DinD, then you'll need to setup a temporary directory and setup write access for either uid:1000 or gid:1000 to it. The latter feels "better" to me. Then we'll end up with this (assuming Gitlab):
zap-baseline:
stage: integration
dependencies: []
allow_failure: true
tags:
- shell
before_script:
- docker pull owasp/zap2docker-stable
- mkdir output; chgrp -f 1000 output; chmod 770 output; cd output
script:
- docker run --rm -v $(pwd):/zap/wrk owasp/zap2docker-stable zap-baseline.py -t http://target:port -J zap-output.json
artifacts:
paths: [output/zap-output.json]
when: always
kilala.nl tags: work, studies,
View or add comments (curr. 0)
2021-02-13 21:31:00
Whelp that's just wonderful.
A QA oversight in Apple's Big Sur updater may lead to your system getting stuck in an endless loop. Worse, if your disk is encrypted using File Vault, you're quite completely hosed. Excellent explanation over here by Mr Macintosh.
Yet again this is a reminder to Always Make Backups!!!
So what's this little mixup Apple made in their quality assurance? The Big Sur updater does not check that it has enough storage space available on your Mac to complete the OS installation. Depending on how much space you have it will either start but refuse to complete the install, or it will start and fail to complete the install. In the latter case, you're in trouble.
With two of our Macbooks Air the install went fine, but Marli's MBA was the smaller 128GB SSD model. With 39GB free space things went tits-up. Thank ${Deity} that we hadn't enabled File Vault on this one.
Now I can at least boot into recovery mode. Disk Utility refuses to properly image the internal storage to a USB drive, but at least dd still works. Man, this is not how I expected my Saturday evening to go. ( ; =_=)
View or add comments (curr. 0)
2021-02-07 22:08:00
I'll talk about it in more detail at a later point in time, but I'm about a week's worth into the Certified DevSecOps Professional training by Practical DevSecOps. So far my impressions are moderately positive, more about that later.
In the labs we'll go through a whole bunch of exercises, applying a multitude of security tests to a Gitlab repository with a vulnerable application. Most of the labs involve nVisium's sample webapp django.nV.
Having reached the half-way point after that one week, I had not encountered two crucial parts of the DevOps / CICD pipeline which I'm not at all familiar with. We're applying all kinds of tests, but we never did the steps you'd expect before or after: creating the artifacts, deploying and running them. As I've said before, I'm #NotACoder.
Instead of focusing on one of the next chapters, today I spent all day improving my Gitlab and Docker install by applying all the required trusts and TLS certificates. This, in the end, enabled me to create, push, pull and run a Docker image with the django.nV web app.
If anyone's interested: here's my Dockerfile and gitlab-ci.yml that I'd used in my homelab. You cannot just throw them into your own env, without at least changing username, passwords and URLs. You'll of course also need a Docker host with a gitlab-runner for deployment.
Note: The Docker deploy and execute steps show a bad practice, hard-coded credentials in a pipeline configuration. Ideally this challenge should be solved with variables or even better: integration with a vault like Azure Vault, PasswordState or CyberArk PasswordVault. For now, since this is my homelab, I'll leave them in there as a test for Trufflehog and the other scanners ;)
kilala.nl tags: work, sysadmin, studies,
View or add comments (curr. 0)
2021-02-07 19:45:00
My homelab runs its own PKI and most servers and services are provided with correct and trusted certificates. It's a matter of discipline and of testing as close to production as possible.
Getting Gitlab on board is a fairly okay process, but takes a bit to figure out.
So my quick and dirty way of getting things set up:
Now, you also want Gitlab and your runners to trust your internal PKI! So you will need to ask your PKI admin (myself in this case) for the CA certificate chain. You will also need the individual certificates for the root and intermediary PKI servers.
Don't forget to restart Gitlab itself, the runners and Docker after making these config changes!
You can then perform the following tests, to make sure everything's up and running with the right certs.
kilala.nl tags: work, sysadmin, studies,
View or add comments (curr. 0)
2021-02-06 21:59:00
Durning the CDP class, one of the tools that gets discussed is Trufflehog. TLDR: yet another secrets scanner, this one built in Python.
I ran into an odd situation running Trufflehog on my internal Gitlab CICD pipelines: despite running it against the intentionally vulnerable project Django.nv, it would come back with exit code 0 and no output at all.
Why is this odd? Because it would report a large list of findings:
But whenever I let Gitlab do it all automated, it would always come up blank. So strange! All the troubleshooting I did confirmed that it should have worked: the files were all there, the location was recognized as a Git repository, Trufflehog itself runs perfectly. But it just wouldn't go...
I still don't know why it's not working, but I did find a filthy workaround:
trufflehog:
stage: build
allow_failure: true
image: python:latest
before_script:
- pip3 install trufflehog
- git branch trufflehog
script:
- trufflehog --branch trufflehog --json . | tee trufflehog-output.json
artifacts:
paths: [ "trufflehog-output.json" ]
when: always
If I first make a new branch and then hard-force Trufflehog to look at that branch locally, it will work as expected.
kilala.nl tags: work, sysadmin, studies,
View or add comments (curr. 0)
2021-02-06 19:35:00
As part of my studying for the CDP course, I've expanded my homelab with a private instance of Gitlab. I've got to say: I like it! A lot. It's good software!
To accomodate my builds I expanded the RAM on my Docker host VM and set up three "gitlab-runners" to pick up jobs from Gitlab CICD pipelines. Microsoft's documentation is outstanding: the runners were installed and configured within minutes.
The only thing I really disliked was their instructions to "wget https://some-url | bash -". That always feels so fscking scary.
As part of my change management process the Docker host of course needed a reboot, to see if things some up correctly. They did and the "gitlab-runner" process was there as well. But it wasn't picking up any jobs! Only when I SSHd into the host and ran "sudo gitlab-runner run" would jobs start flowing.
At first I thought I just didn't understand the concept of the runner process well enough. Maybe I hadn't set them up correctly? Then I decided to do the logical thing: check the logs. I've been teaching my students to do so, so why didn't I? :D
"sudo systemctl status gitlab-runner -l" showed me the following:
$ sudo systemctl status gitlab-runner -l
â— gitlab-runner.service - GitLab Runner
Loaded: loaded
Active: active
...
Feb 06 19:24:37 gitlab-runner[20361]: WARNING: Checking for jobs... failed
runner=REDACTED status=couldn't execute POST against https://REDACTED/api/v4/jobs/request:
Post https://REDACTED/api/v4/jobs/request: x509: certificate signed by unknown authority
The self-signed cert isn't too surprising, since I still have a backlog item to get that fixed. I wanted to first get the basics right before getting a proper cert from my PKI. But I thought I had dealt with that by registering the runner with a CA cert override.
Checking "/etc/gitlab-runner/config.toml" showed me where I had gone wrong: the CA cert override path was relative, not exact.
[[runners]]
name = "REDACTED"
url = "https://REDACTED"
token = "REDACTED"
tls-ca-file = "./gitlab.pem"
executor = "docker"
I had assumed that the cert would be picked up by the runner config and stored elsewhere, instead of being referenced from the file system. Wrong! I made sure to copy the self-signed cert to "/etc/gitlab-runner/gitlab.pem" after which I corrected the "config.toml" file to use the correct path.
One quick restart of the runner service and now jobs are automatically picked up!
kilala.nl tags: work, studies, homelab,
View or add comments (curr. 3)
All content, with exception of "borrowed" blogpost images, or unless otherwise indicated, is copyright of Tess Sluijter. The character Kilala the cat-demon is copyright of Rumiko Takahashi and used here without permission.