Sorry for the Rough Ride

Hi there. As you surely noticed the site was down for the last couple of days. This was due to a chain of events that in the end resulted in a complete outage. The system the site is hosted on is running from a dedicated server and the data resides on a simple RAID-1 array. On Monday one of the drives in that array died and so the site went down for the first time. Using the rescue system I was able to create a current backup (in addition to the ones that are created automatically) and ordered the failed drive to be replaced. This was done but while restoring the array the second drive died from the stress of rebuilding. With the backup at hand I ordered the second drive to be replaced as well, screw the data on the disks and rebuild the array from scratch. Now, this process didn’t go as smooth as I hoped because the major part of the following days was spent with convincing the hoster that the new replacement drive couldn’t be bothered to join the array. Part of the delay of course was due to the fact that I need to not only go to work but also do some actual work there so that I couldn’t be immediately available for follow-up questions from the hoster which mostly consisted of “Are you really sure that we should …” and my simple answer “Yes, I am” (felt like being at an elaborate wedding at some times).
Anyway, one the array was up with a finally again replaced second drive and the OS installed I could go about reinstalling the bits and pieces. And finally I was able to restore the data which took a long, long time for I had to use my meager 2 Mbit/s uplink to transfer roughly 70 GB of data. That’s why it took so long to bring the site up again.
I heard that some of you were encountering troubles throughout the outage, mainly due to some services seemingly being dependent on some response from the site. My apologies for that, I’ll look into those reports and will try to make sure this stuff is removed or at least changed so that it will fail gracefully should another outage occur.
On the plus side I took the chance and went through all the backup systems and refined some of those to be able to get the site up faster. Still, the major blocker here is my upload speed and there’s nothing I can do about it in the near future.
That all said, I again apologize for any inconvenience and hope you still enjoy your ReadyNAS and my add-ons.

Bye-bye Slack!

Well, switching to Slack as the primary support channel didn’t work out either. Actually, I did work perfectly for the first couple of days. And then the spammers found the invite form and from there everything went downhills. I can’t restore the invite functionality for all invites have been used up and there’s no way to get them back. So I’ll close the Slack channel again and check out other options. For the time being there’s still email but given the amount of mail I receive daily due to my job there’s a good chance I’ll overlook a message or simply forget about it at the end of the day.

A side note to all those who complained that there is no email address to send support mail to: That’s not true, you for some or other reasons just didn’t try to simply use the “reply” function of your email client. Additional hint: There’s no “noreply” type mailbox here.

Bye-bye Forums!

As the headline says: I got rid of the forums on this site. Not because I don’t want to hear about problems with the add-ons anymore but because the forums never really worked. Not for me and not for the users. I didn’t get notified of new or updated posts. Users weren’t able to access the forums. Not good. So I closed the forums for good. And decided that Slack might be a better alternative. The catch: you need to get an invitation to be able to join the RNXtras team on Slack. But fear not, all you need to do is to sign up using this link:

  • Add me to rnxtras.slack.com (link removed thanks to spammers)

After entering your email address into the form (no, I won’t sell nor even do anything with it, it’s just needed by Slack) you’ll get an invitation and you will be able to join the RNxtras team on Slack and get a chance to actually talk to me. But please don’t expect to me to be present 24/7. I’m a normal working guy with a family and it’d be really nice if you could keep that in mind.

 

Week of the overdue updates

After having finished two moves – one with the company, one private – and after rebuilding the development environment in the new location I finally found the time to run upgrades on various add-ons for the different ReadyNAS platforms. The main focus for the coming updates is on Python and the apps depending on this programming language. Some of you are painfully aware of the fact that since Python 2.7.9 the handling of SSL connections has changed. This alone wouldn’t have been a real problem since most apps already dealt with that change properly, SABnzbd being the sole exception I know of.

However, thanks to various vulnerabilities in the SSL protocol stack SSLv3 support was dropped by most web sites and certificate issuers. This posed a real problem since the OpenSSL library shipping with the older ReadyNAS models running RAIDiator 4.x and 5.x doesn’t properly the newer protocols TLS 1.1 and TLS 1.2 that are now in widespread use. The only way to fix this was to backport OpenSSL 1.0.2 to the older ReadyNAS platforms and do a complete rebuild of the Python packages. This took some trial and error and of course some time for compiling and testing. This being done I can now start to update the apps and see what has changed there.

In a nutshell, here’s what will be released over the next few days:

  • Python 2.7.10 with fixed SSL handling for ReadyNAS OS 4 (x86 and Sparc) and ReadyNAS OS 5 (ARM)
  • CouchPotatoServer for all ReadyNAS platforms
  • SickRage for all ReadyNAS platforms
  • Syncthing for all ReadyNAS platforms except ReadyNAS 4 Sparc
  • Deluge for all ReadyNAS platforms except ReadyNAS 4 Sparc
  • NZBGet for all ReadyNAS platforms
  • OwnCloud 8.2.2 for all ReadyNAS platforms

and possibly some other stuff I forgot to mention. In parallel I’m also working on another project I may write more about in an other post. For the time being I’m busy enough with preparing the updates.

Apologies for “spamming

As most of you surely have noticed by now the forum software used on my site went nuts and started to send out notifications for new posts to each and every user of the site. I fully understand that this is totally annoying and thus I also have full understanding for the anger this has caused. Let me assure you that this was totally unintentional and that I’m deeply sorry.

What seems to have happened is that after installing an update for the forum plugin the notification settings were reset in such a way that all users got notified about every new post or reply. Unfortunately there was one sole exception: me. I didn’t receive any of these messages, otherwise I’d have stopped this way earlier. The next unfortunate thing that happened was that while users started to complain in the forums, no one did send me an email to inform me about the problem. Since I wasn’t informed about the forum posts about the matter either, again, I didn’t notice anything. That I’m currently on Christmas vacation didn’t help either, otherwise I might have noticed the increased number of site visits and would have started to investigate the reason. Oh well.

So it wasn’t before yesterday when I finally received an email about the problem that I could go about fixing the problem. Until then a lot of mails had already queued up and were on their way. Again, I’m totally sorry for that.

Now the good news is that I believe that the problem is fixed now. I deleted the forum that I think caused most of the problems and I also cleared out all the messages that were still waiting for delivery on my server. I disabled the sending of notification emails and I’m going to check the flags for every user and disable notifications and subscriptions to forum topics there. I have also set up active monitoring of the mail queue size and delivery rate so that I should now get an automatic notification if the site sends out an unusual amount of mails. So in case this happens again (hopefully not) I’ll get informed earlier and can take immediate action. I’m currently watching the mail queues and as far as I can tell everything is back to normal. If things start to look strange again I will completely disable and probably delete the forums.

Lessons learned:

  • Don’t trust plugin updates
  • Monitor more
  • Monitor even more

I wish all of you a Merry X-Mas, hopefully without any further disturbances from my site, a peaceful time and a Happy New Year,

Stefan