Synology vagueries: slow transfers, 100% volume util, very high load average, very high IOWAIT

2018-06-28 22:30:00

I've been a very happy user of Synology systems for quite a few years now. The past few weeks I've ran into quite some performance issues though, so I decided to get to the bottom of it.

Symptoms:

• CPU usage is low, with <15% active usage.
• CPU usage is stuck on 70-85% IOWAIT.
• My load average is very high, at 14.xx consistently.
• The performance monitoring tool shows 100% volume utilization.
• Very low transfer speeds to/from the NAS.
• Very long connection times on the file shares.
• Very slow logins and bad response times on SSH and the web interface.

I have undertaken a few steps that seem to have gotten me in the right direction...

1. I have gone over the list of active services and disabled the ones I do not use.
2. I verified the installed packages and I've removed all the things I really don't need.
3. I have disabled the Universal Search function, which cannot be disabled without trickery (see below).
4. I have disabled the Indexing daemon in full, which also cannot be disabled without extra effort (also below).

In order to disable Universal Search:

1. Login through SSH
2. cd /var/packages/SynoFinder
3. sudo cp INFO INFO.orig
4. sudo vi INFO

Make the following changes:

ctl_stop="yes"
ctl_uninstall="yes"

You can now restart Package Center in the GUI, browse to Universal Search / SynoFinder and stop the service. You could even uninstall it if you like.

In order to disable the Indexer daemon:

1. Login through SSH
2. sudo synoservice --hard-stop synoindexd
3. sudo synoservice --disable synoindexd

The second step is needed to also stop and disable the synomkthumb and synomkflvd services, which rely upon the synoindexd.

One reboot later and things have quieted down. I'll keep an eye on things the next few days.

kilala.nl tags: sysadmin,

View or add comments (curr. 5)

Keywords for this week: Windows, Linux, PKI and DAMTA

2018-06-24 20:41:00

It's gonna be a busy week! 

Most importantly, I'll be taking CQure's "DAMTA" training: Defense Against Modern Targeted Attacks. Basically, an introduction to threat hunting and improved Blue Teaming. Sounds like it's going to be a blast and I'm looking forward to it a lot :)

Unfortunately this also means I'll be gone from the office at $CLIENT for three days; that bits, 'cause I'm in the midst of a lot of PKi and security-related activities. To make sure I don't fall behind too much I'm running most of my experiments in the evenings and weekend. 

For example, I've spent a few hours this weekend on setting up a Microsoft ADCS NDES server, which integrates with my Active Directory setup and the base ADCS. My Windows domain works swimmingly, but now it's time to integrate Linux. Now I'm looking at tools like SSCEP and CertMonger to get the show on the road. To make things even cooler, I'll also integrate both my Kali and my CentOS servers with AD. 

Busy, busy, busy :)

kilala.nl tags: work, sysadmin,

View or add comments (curr. 0)

Handy tool to troubleshoot your Microsoft ADCS PKI

2018-06-23 14:08:00

It has been little over a year now since I started at $CLIENT. I've learned so many new things in those twelve months, it's almost mindboggling. Here's how I described it to an acquaintance recently:

"To say that I’m one lucky guy would be understating things. Little over a year ago I was interviewed to join a project as their “pki guy”: I had very little experience with certificates, had messed around a bit with nShield HSMs, but my customer was willing to take a chance on me. ... ... A year onwards I’ve put together something that I feel is pretty sturdy. ... We have working DTAP environments, the production environment’s been covered with a decent keygen ceremony and I’m training the support crew for their admin-tasks. There’s still plenty of issues to iron out, like our first root/issuing CA renewal in a few weeks, but I’m feeling pretty good about it all."

As I described to them, I feel that I'm at a 5/10 right now when it comes to PKI experience. I have a good grasp of the basics, I understand some of the intricacies, I've dodged a bunch of pitfalls and I've come to know at least one platform.

How little I know about this specific platform (Microsoft's Active Directory Certificate Services) gets reinforced frequently, for example by stumbling upon Brian Komar's reply to this thread. The screenshot above might not look like much, but it made my day yesterday :) "Pkiview.msc" you say? It builds a tree-view of your PKI's structure on the lefthand side and on the right side it will show you all the relevant data points for each CA in the list. 

This is awesome, because it will show you immediately when one of your important pieces of meta-data goes unavailable. For example, in the PKI I built I have a bunch of clones of the CRL Distribution Point (CDP) spread across the network. Oddly, these clones were lighting up red in the pkiview tool. Turns out that the cloning script had died a whiles back, without any of us noticing. 

So yeah, it may not look like much, but that's one great troubleshooting tool :)

kilala.nl tags: work, sysadmin,

View or add comments (curr. 0)