Page 1 of 2

Maintenance

Posted: Sat Feb 25, 2017 12:47 pm
by Dbug
I'm going to stop the access to all the servers in a couple of hours, and this is probably going to be for a good part of the weekend because I'm planning to do a complete evaluation of how things are, including operating system upgrade, drive changes, backups verification, etc...

In the meantime there will be a secondary server serving a status page to show what is happening.

I know it's annoying, but it's probably less annoying than these freezes that happen now and then.

Re: Maintenance

Posted: Sat Feb 25, 2017 7:44 pm
by Dbug
So, end of Phase 1 (replacing the two old drives by one new drive):
15:00 - Server offline and unplugged
15:30 - Disk #2 content have been copied onto Disk #1
16:00 - Disk #2 is now replaced by a new and much larger SSHD drive
16:30 - Formating the SSHD to Ext3 format
16:40 - Copying the content of Disk #1 to the newly formated SSHD
18:50 - ...still copying...
19:10 - Done copying
19:15 - Did some fstab magic to point the old drive to the new drive and rebooted the system
19:39 - Switched back the port forwarding to Miniserve
I'm done for today, but I'm going to continue tomorrow, this time this will be the operating system upgrade/fix (including php and phpbb to the latest version), and then reworking all the backup processes.

Normally we should not have lost anything, and all sub-domains should be up and running, and SVN(*) should be ok as well.

You should be able to use everything now until sometimes tomorrow morning when I will start messing with stuff again :)

Not sure how many of you actually saw the "work in progress" log, I tried to maintain it while stuff was happening.

Would be interesting to see if the freezes are gone, if that's the case then it was probably one of the harddrives having issues.



(*) anyone trying to put one of the SHA1 pdf on the SVN depot will be dealt with extreme prejudice ;)

Re: Maintenance

Posted: Sat Feb 25, 2017 9:28 pm
by Symoon
Thanks Dbug, as always.
It seems the forum is reacting perfectly now! :D

Re: Maintenance

Posted: Sat Feb 25, 2017 9:35 pm
by Chema
Yep, thanks indeed. I followed the progress a bit (I saw 3-4 updates). Nice touch!

By the way SHA1 pdf ?

Re: Maintenance

Posted: Sat Feb 25, 2017 9:57 pm
by Dbug
Chema wrote:Yep, thanks indeed. I followed the progress a bit (I saw 3-4 updates). Nice touch!
By the way SHA1 pdf ?
https://arstechnica.com/security/2017/0 ... ay-follow/

Regarding the performance, I'm not sure why it had got better.
Technically compared to before the maintenance what has changed is:
- The server is not in the server room anymore (it's on my desk), so different power, ethernet cable, no UPS, etc...
- The files have been moved from the old 750 GB drive to a new 2TB SSHD

I guess that could have been some hardware issue, maybe the HDD on the brink of dying?
I was running the performance tool to see if there was any spikes in CPU when the server was not answering, but from a UI point of view everything seemed fine.

Re: Maintenance

Posted: Sat Feb 25, 2017 11:07 pm
by Chema
Haha I knew about the collision, but not about the colliding pdf examples breaking anything. :)

Re: Maintenance

Posted: Sun Feb 26, 2017 7:45 am
by coco.oric
Thanks dbug
It seems that it works fine. This week, i was searching some code keyboard for a friend and several times, my computer reports me that the internet adress was unavaible

Re: Maintenance

Posted: Sun Feb 26, 2017 10:42 am
by Dbug
I'm going to start the second stage in 15 minutes.

Re: Maintenance

Posted: Sun Feb 26, 2017 7:33 pm
by Dbug
Testing if we can actually write stuff.

Re: Maintenance

Posted: Sun Feb 26, 2017 7:59 pm
by Dbug
So, the second stage is done, which means we now have the following improvements:
- The system disk was upgraded from a (getting old) Intel X25 to a Samsung 540 SSD
- The storage drives have been upgraded from a standard (and getting old as well) 7200 RPM 750GO HDD to a 2TB SSHD
- Operating system upgraded from a broken Ubuntu 12/14 to a fully up-to-date 16.04 LTS
- PHP upgraded from version 5 to 7
- PhpBB upgraded from version 3.0 to 3.2 (we have unfortunately lost the skins, will have to be redone)
- DokuWiki had not been upgraded since 2006, it's now running the very last version

For the timeline of the events:
11:00 - Server offline and unplugged
11:10 - Installed the new SSD
11:15 - Full update of the OS (Ubuntu 16.04 LTS)
11:50 - Did a new backup of the phpbb database
13:20 - Fighting with php, samba and mysql
13:44 - Trying to get webdav/svn up and running
14:25 - mmuman confirmed that the SVN access was working
14:55 - Still fighting with phpbb which is not happy with my version of php and mysql
16:10 - Only took 3 hours to figure out that I should have replaced $dbms = 'mysql'; by $dbms = 'mysqli'; in the config.php file...
16:20 - Updated from phpbb 3.0.12 to 3.1.10 (lost all my visual configuration... snifff... :'(')
16:40 - Updated from phpbb 3.1.10 to 3.2.0
17:00 - Tried to get the styling sorted, and failed, so it's going to stay default blue for a while
17:20 - Working on adding the crontab scripts for the dynamic dns
17:45 - Updating Doku Wiki to the latest stable release. It's not like if it was 10 years old...
19:00 - Correctly setup the .htaccess apache2 configuration
19:30 - Time to rescrew everything
We are running without any backup script, I did not have time to do that, but hopefully everything should be running.

So your mission is now to go around and poke at everything to try to find what I forgot, what is broken or missing, being pictures, messages, pages, redirects, subdomains, etc... the faster we find stuff the better it will be.

Thanks for your patience :)

Re: Maintenance

Posted: Sun Feb 26, 2017 8:08 pm
by Chema
Great! Thanks Dbug. I will have a look around..

Edit: found that the tabs at the defence-force.org site (the new one) no longer work. Games, news, forum...

Re: Maintenance

Posted: Sun Feb 26, 2017 8:50 pm
by Dbug
Edit: found that the tabs at the defence-force.org site (the new one) no longer work. Games, news, forum...
Indeed, looks like I broke something when I was editing the virtual hosts.

Is it a blocker, or can we live with that for a week or so?

Re: Maintenance

Posted: Sun Feb 26, 2017 8:59 pm
by ibisum
Thanks for the huge effort to keep us all updated and the systems all on course .. looking pretty good so far. The only thing I miss is the "New Posts.." link that used to be available all over the forums, but I guess I can set up smart filters with the new phpBB .. anyway, thanks again for the work! defence-force.org is a great destination every week ..

Re: Maintenance

Posted: Sun Feb 26, 2017 9:08 pm
by Chema
I don't think it is a problem, Dbug. We can live without that for a couple of weeks..

Ibisum, you have a New posts link in the quick links menu on the top left.

Re: Maintenance

Posted: Sun Feb 26, 2017 9:23 pm
by ibisum
Found it, thanks for that! :)