Adventures in web community infrastructure

For anything that doesn't fit the other forums.

Moderator: TMAX

User avatar
stuartcnz
Site Admin
Posts: 873
Joined: Fri Dec 19, 2008 8:05 pm
Location: Aotearoa, New Zealand
Contact:

Adventures in web community infrastructure

Post by stuartcnz »

Given the ups and downs that this forum has experienced, on the technical side. I have decided that it needs to be rebuilt from the ground up, under a completely new architecture. This thread is for documenting the journey. It maybe a long ride; probably with some side excursions, and it will definitely be bumpy. Progress, and even any activity will be sporadic.

Buckle up, and enjoy the ride.
User avatar
somewhereinusa
Posts: 322
Joined: Sat Feb 12, 2011 2:44 pm
Location: Andrews, Indiana
Contact:

Re: Adventures in web community infrastructure

Post by somewhereinusa »

Thanks for all you are doing, I like this forum because it seems to be people who enjoy doing instead of buying.

Dick
User avatar
stuartcnz
Site Admin
Posts: 873
Joined: Fri Dec 19, 2008 8:05 pm
Location: Aotearoa, New Zealand
Contact:

Re: Adventures in web community infrastructure

Post by stuartcnz »

To start with, a bit of forum infrastructure history is in order. (part one)

In the beginning, there was a website called mrsharkey.com. This web site was "The Net’s largest compendium of information relating to House Buses and House Trucks." In 2004 Sharkey decided to add some software to his site, in order that he could do some blogging like activity. The software he chose, was a forum software called PHPBB2. And that was the start of the forum, that we have today.

Over the next seven years, Sharkey cultivated the forum, kept the software updated, and made large modifications to the code, which allowed it to be very open, and at the same time spam free. At the end of this time, the customizations were so significant, that the software could no longer be updated, through the normal PHPBB2 method.

By this time, Sharkey had come to the conclusion, that he no longer had enough reason to keep maintaining the forum, which is when I became the custodian. Sharkey spent about a month, helping with the transition to my infrastructure, which is when we upgraded to using PHPBB3. To do this, meant we lost all of Sharkey's customizations, so the way I have operated the forum has necessarily been with more restrictions in place, to be able to maintain a lack of spam activity.

When I first started maintaining the forum, I was hosting it on a Hostgator reseller account. This was easy for me to do, as I already had the account, and they took care of general server security. They also have automated email management and various other goodies, which made it a good working solution at the time. This ease of use, does however come with some usage restrictions, which over time became quite bothersome to me, as it restricted my own development of other web based projects.
Stealth Camper
Posts: 792
Joined: Fri Apr 09, 2010 5:17 pm
Location: Oklahoma
Contact:

Re: Adventures in web community infrastructure

Post by Stealth Camper »

Ok...I am just gonna kinda probably throw questions in from time to time about the process...

PHPBB was one you already answered - that it is robust, current, and remains supported.

Web hosting... I have heard of Hostgator and in the world of hosting, seems to be in the middle of the pack in size of web hosts. I have been thinking about getting a site, and after looking at quite a few, am still at least mildly bewildered for the most part. While reading about each host, I see things that I like and feel would be good, things I know I don't care about right now like internet sales, and then a bunch of stuff that just doesn't compute... I have looked at the list of biggest, read reviews, compared prices, and in general am just about ready to choose based just on which name I like best...

Can you elaborate on your selection process for your hosting needs? Maybe bring me back to mere chaos from the madness that is the web hosting selection topic?

And are you going to stay with Hostgator, or change?
User avatar
stuartcnz
Site Admin
Posts: 873
Joined: Fri Dec 19, 2008 8:05 pm
Location: Aotearoa, New Zealand
Contact:

Re: Adventures in web community infrastructure

Post by stuartcnz »

To answer your question backwards, I haven't been using Hostgator for quite some time.

If you haven't run a web site before, they are a reasonable choice. I found up time to be good, as was customer service.

Any of those type of hosts, are going to be on shared hosting plans, which means computing resources can be tied up, slowing things down.

If you want full freedom, then you need either a dedicated server, or a VPS (virtual private server) which guarantees server resources. The down side is you need to know how to set up and maintain a server, before you can even start to look at setting up a web site. There is a LOT to learn about internet security, before you go down this path.
User avatar
stuartcnz
Site Admin
Posts: 873
Joined: Fri Dec 19, 2008 8:05 pm
Location: Aotearoa, New Zealand
Contact:

Re: Adventures in web community infrastructure

Post by stuartcnz »

forum infrastructure history. (part two)

For some other internet projects, I wanted to try some Python based software. Unfortunately many shared hosts don't allow you to do that, or make it difficult to do so.

I discovered linode.com which offers their cheapest service at US$10 per month, for a VPS. For an extra US$2.50 per month, per VPS, they also provide a comprehensive backup service, for your VPS's. As the Hostgator service I was using was costing US$25 per month, I realised that two Linode VPS's, would be the same price as what I was paying for Hostgator. So that is what I did. I set up one linode VPS as an email server, and another as a web server. Initially I set up both, with CentOs7 Linux as the underlying operating systems, as it has a ten year service life, and is an enterprise grade operating system. This was particularly important for the email server, as email is very tricky to get right, and I didn't want to be setting up a new system every couple of years, given how difficult and time consuming that can be.

I have found Linode to be very good for me. I made an advance payment, which keeps me in credit, for two servers, for about another three years.

A while later I was told about another company which works on a completely different business model, called cloudatcost.com, where they offer plans on a one off cost basis. They were reputed to be somewhat unreliable, but very cheap for what they offer, so a good choice as development servers. I currently have available with them, 10 CPU's, 9GB ram, and 140GB of disk space, to set up different VPS's with. I can get more for free, if someone signs up to one of their plans and use the email address that I signed up with as a reference.

Eventually I converted all of my other PHP based websites, until the only PHP based website left running was the forum. At that stage I had set up a third Linode VPS, (this time with Debian Stable Linux as the operating system, which is what I use on my desktop) for my Python based stuff and found it difficult to justify having a third Linode VPS, just to host the forum, so I shifted it to a VPS on cloudatcost, which as mentioned has no recurring cost. It is since the forum has been at cloud at cost, that the outages have occurred, and I also believe the forum is sometimes less responsive than when it was on Linode.

This brings us up to the present: Which is where I want to convert the forum to run on Python, so I can shift it back to the much more resilient and reliable Linode service, on my existing webserver there.
User avatar
stuartcnz
Site Admin
Posts: 873
Joined: Fri Dec 19, 2008 8:05 pm
Location: Aotearoa, New Zealand
Contact:

Re: Adventures in web community infrastructure

Post by stuartcnz »

The first step in the journey to a new forum infrastructure, is to make the existing forum as resilient as possible. To that end I am working towards automating database backups, that should also be able to automatically email them to me; so that if the server fails again, I will have an up to date database to rebuild from.

So far I have been able to assemble a php script, to make the backup. Also I have separately worked out how to email a file from the server, to myself. now I need to workout how to insert the email code, into the php script, in a way that will automatically send the latest backup. Then it will just be a matter of setting up a cron job, to run the script every day.
User avatar
stuartcnz
Site Admin
Posts: 873
Joined: Fri Dec 19, 2008 8:05 pm
Location: Aotearoa, New Zealand
Contact:

Re: Adventures in web community infrastructure

Post by stuartcnz »

Database, backup and emailing script is working now. I just have to set up a cron job, to make it run automatically.

A copy of the script is here: https://kallithea.ethicallogistics.com/_admin/gists/2 and should work for any mysql databases.
User avatar
stuartcnz
Site Admin
Posts: 873
Joined: Fri Dec 19, 2008 8:05 pm
Location: Aotearoa, New Zealand
Contact:

Re: Adventures in web community infrastructure

Post by stuartcnz »

Cron job has been written. Will see if I got that part right, in the morning. I have scheduled the backup to run at one minute past three in the morning everyday. If it works, I should have a new backup in my inbox when I wake up. :)

cron job is written as

Code: Select all

1 3 * * * /usr/bin/php /path/to/sqlbackup.php
User avatar
stuartcnz
Site Admin
Posts: 873
Joined: Fri Dec 19, 2008 8:05 pm
Location: Aotearoa, New Zealand
Contact:

Re: Adventures in web community infrastructure

Post by stuartcnz »

First thing that I do when setting up a public facing server, is to harden it for security. I use the following page as a guide, though do things a bit different in various parts. https://www.linode.com/docs/security/se ... our-server

If I want to run anything that runs on PHP, such as this forum, a wordpress.org site, Friendica, etc. Then I need to set up the server as a LAMP (Linux, Apache2, Mysql, PHP) server. For that I find this to be a quick and easy guide. https://www.linode.com/docs/websites/la ... n-8-jessie

General rule of thumb, is that I install the least amount of stuff, to run what I want on a server. For python based stuff, it starts with a web server, which I use Nginx for. Python is already installed by default on all Linux servers. If I need to run a database, I will generally use Postgresql. This setup would be called a LEPP server (Linux, Nginx pronounced "engine x", Postgresql, Python)
User avatar
stuartcnz
Site Admin
Posts: 873
Joined: Fri Dec 19, 2008 8:05 pm
Location: Aotearoa, New Zealand
Contact:

Re: Adventures in web community infrastructure

Post by stuartcnz »

Stealth Camper, I should probably mention at this point, that when you start the process of setting up a web page; it is a good idea to register your domain name with a different company, to the one that will be hosting your site. That way, when the time comes, that you wish to shift your site (will probably happen some time) to a different host. The hosting company can't hold your domain name hostage. It has been known to happen.
User avatar
stuartcnz
Site Admin
Posts: 873
Joined: Fri Dec 19, 2008 8:05 pm
Location: Aotearoa, New Zealand
Contact:

Re: Adventures in web community infrastructure

Post by stuartcnz »

Automatic backups are working! :) Latest backup was sent to my email inbox at 03:01 this morning.
User avatar
stuartcnz
Site Admin
Posts: 873
Joined: Fri Dec 19, 2008 8:05 pm
Location: Aotearoa, New Zealand
Contact:

Re: Adventures in web community infrastructure

Post by stuartcnz »

I have just setup and hardened a test server for developing the new forum on. It will be visible at forum.nomadicista.org, once the dns propagates, about this time tomorrow. I haven't opened the firewall yet, so it still won't be publicly reachable until I do that, probably sometime tomorrow. And then I will also get the HTTPS certificate too. Might as well start on the right foot. :)
Dusty
Posts: 356
Joined: Sat Jan 09, 2010 6:06 am
Location: New Zealand
Contact:

Re: Adventures in web community infrastructure

Post by Dusty »

I've got no idea what you just said Stuart , but it sounds bloody awesome :)

Many thanks for your time and hard work .

Regards
Last edited by Dusty on Mon Aug 29, 2016 3:09 pm, edited 1 time in total.
58 Bedford House
Stealth Camper
Posts: 792
Joined: Fri Apr 09, 2010 5:17 pm
Location: Oklahoma
Contact:

Re: Adventures in web community infrastructure

Post by Stealth Camper »

Awesome!! Thanks so much!
Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests