Comic Genesis Forums

Mycohl wrote:No. Completely factual, accurate and an answer to the question asked.

Actually, Neo's answer is a better choice, because then the index.html file is one that has actually already been THROUGH AutoKeen, even though the server is down. Then the ads and all the tags work. ^_^

Mycohl wrote:Since we are being condescending and nitpicking, that doesn't conform to XHTML/XML. You need to close it with />.

Who was being condescending? He was merely informing people how to change their index.html files if the server is down, and how to bypass AutoKeen, but still have tags. It's not nitpicking to tell people how to do it right when doing something wrong could hose the author's comic.

Mycohl wrote:

Kisai wrote:No instead a breif outage was caused by resyncing replication, which requires locking the tables, and hence locking apache out of writing logs. I've made a few modifications to the mysql logging wrapper so it just throws away the log entry if it can't reconnect to the database, that should stop apache from freezing solid, but that won't do anything for autokeen, ftp and siteadmin if it's mysql that's being broken.

Why is that mysql being broken? Autokeen should sleep if it can't touch the db for however long it takes to sync and unlock; if that is what is happening, I wouldn't call that broken. I haven't experienced ftp or siteadmin blocking at all, even when CG's httpd was hung/down. I'm sorry I can't be of any help.

Okay, here's a technical run down

MySQL -> Apache Logging
|--AutoKeenDispatcher->AutoKeen
|--FTP
|--Siteadmin8080(Via Boa)
|--Siteadmin (via Apache)-|-Webstats page
|--Logsplitting->Webalizer-|

MySQL is core to all the services operating.
That said...
If MySQL is down, or race condition what is supposed to happen:
Apache is supposed to just keep on chugging provided it's not a disk error, I switched it from always retry, to give up after one retry in the logging script.
Unfortunately various bugs crop up when things are updated, which is why MySQL has not been updated to 5.x, because you can't resync replication anymore except by hotcopy, which is, dumb. I have a feeling that resyncing replication is buggy and that's why the depreciated the feature, however that is so not what was being run at 4:55AM

AutoKeen can not and will not work without mySQL running, and if mySQL is kill'ed for any reason, it trashes the tables, which is why I pull my hair when someone touches mySQL, because I then have to pull the replication from the other server, pain in the butt.

My apologies Mycohl, it was not my intent to put you down or make it seem like your reply was invalid.

However, having gone back and re-read the posts I realise that I misread your post. For some reason I thought you were saying to delete the template file from the wokspace folder, which you clearly did not do.

My bad...

Queue seems to be stalled at 3043 updates.

...and it's my comic's anniversary today, too, and I have a birthday strip and everything.

Kisai: Why can't you get a new server? My mental image is that you just go to a merchant that sells servers, and buy one. Is there a lack of money? (Could we hold a fundraiser?) Or is there some reason why getting a server is more complicated than that?

SailorPtah wrote:Queue seems to be stalled at 3043 updates.

...and it's my comic's anniversary today, too, and I have a birthday strip and everything.

Kisai: Why can't you get a new server? My mental image is that you just go to a merchant that sells servers, and buy one. Is there a lack of money? (Could we hold a fundraiser?) Or is there some reason why getting a server is more complicated than that?

Yes, it says 3043 and I need to update my comic too!!

Congratulations on teh comicy-anniversary-goodness too....With a little bad mixed in because of the stalled queue...

The pubes are back... I think everyone needs to start shaving down there.

Waaaay ahead of you.

Dr Neo Lao wrote:I realise that I misread your post.

No worries. I, in turn, apologize for jumping to a conclusion and replying in a snarky manner.

Kisai wrote:if mySQL is kill'ed for any reason, it trashes the tables

Why?? Whenever mysql stops, the tables go away? What happens when you reboot? That's insane and has to be indicative of a more profound problem.

I am sure you are much more experienced than I, and you have the insight of having designed and implemented the system yourself. All I can do is ask the proverbial blind man's inane questions. If you don't want to be bothered, I completely understand.

Why are you logging to mysql? So that you don't have to rotate logs? Does it have something to do with autokeen and how it handles data? If you have webalizer fetching log data from the db, that's going to eat cycles like candy and strain mysql as well. Rather than invest in moving everything to a new server, why not just set up a dedicated mysql box and lock it down? I'm uncertain as to whom you are having security problems with regarding mysql. Is CG's apache/boa/mysql setup being shared with some other group of people? Is the box shared with other keenspace-related services? (I assume it is, which would explain why upgrading is not an option.)

I really don't know who/what "keenspace" is, how many machines/clusters they have or how many they have allocated for CG's use. With as much strain as I imagine CG puts on the db back-end, I can't imagine that running mysql alongside apache, boa, ftp and a continuously forking perl/bash script is healthy. Especially if the machine has intermittent postfix hangs. The only thing that I can (ignorantly) recommend is getting mysql on a separate box/cluster, getting it stable and then getting your dependant services working. With a mysql box dedicated to CG's use, that eliminates both the security problems and (hopefully) the stability problems.

theSuburbanLetdown wrote:Waaaay ahead of you.

LET THE MASS SHAVING BEGIN!!!

Mycohl wrote:

Dr Neo Lao wrote:I realise that I misread your post.

No worries. I, in turn, apologize for jumping to a conclusion and replying in a snarky manner.

Kisai wrote:if mySQL is kill'ed for any reason, it trashes the tables

Why?? Whenever mysql stops, the tables go away? What happens when you reboot? That's insane and has to be indicative of a more profound problem.

I am sure you are much more experienced than I, and you have the insight of having designed and implemented the system yourself. All I can do is ask the proverbial blind man's inane questions. If you don't want to be bothered, I completely understand.

Why are you logging to mysql? So that you don't have to rotate logs? Does it have something to do with autokeen and how it handles data? If you have webalizer fetching log data from the db, that's going to eat cycles like candy and strain mysql as well. Rather than invest in moving everything to a new server, why not just set up a dedicated mysql box and lock it down? I'm uncertain as to whom you are having security problems with regarding mysql. Is CG's apache/boa/mysql setup being shared with some other group of people? Is the box shared with other keenspace-related services? (I assume it is, which would explain why upgrading is not an option.)

I really don't know who/what "keenspace" is, how many machines/clusters they have or how many they have allocated for CG's use. With as much strain as I imagine CG puts on the db back-end, I can't imagine that running mysql alongside apache, boa, ftp and a continuously forking perl/bash script is healthy. Especially if the machine has intermittent postfix hangs. The only thing that I can (ignorantly) recommend is getting mysql on a separate box/cluster, getting it stable and then getting your dependant services working. With a mysql box dedicated to CG's use, that eliminates both the security problems and (hopefully) the stability problems.

As I stated, the machine is dual processor, and running threaded apache, therefor it can not log to the hard drive.

Conventional apache, prefork mode, writes one line at a time, in order, to one master log file, that remains locked the entire time. This is fine on a very small web server where it doesn't fill up fast, but to give you a reference point, 40GB of logs in 5 days. You simply can't rotate logs fast enough to let webalizer process it. Plus they need to be split on a per domain basis.

Second it's also dynamicaly generated virtual hosting, Apache's static virtual hosting tops out at about 300 sites, we have 25 times that. So there is no provision to simply create one log file per site, as it would require shutting down apache for hours to rotate the logs.

Every logging system I tried, was not thread safe, and would eventually hang apache after a while. I eventually just wrote a non-thread-safe logging script that each thread calls and dumps straight to mysql. This solves the problem of log files remaining locked or becoming out of order. MySQL has one table per day, and drops the oldest table on every new day. So this gets rid of the rotating logs issue, and then when webalizer is supposed to run, it's piped through webalizer so no temporary files are created. It really can not be made any more efficient.

Having a physically separate mySQL server is out of the question. The only thing being replicated between the forum server and the main server is the user tables for phpbb and autokeen. None of the logging or anything.

As for why we have not received a new server, we keep asking and we keep getting told 'soon'. There is nothing left to optimize on the existing server, it's getting limited by disk bandwidth.

thank you for that, Kisai. i appreciate you taking time to explain things clearly. Let's all hope "soon" comes soon.

Just wondered how it was all going with the fix? - Is it a big job?

We are down AGAIN!

This is getting #V<%!^& ridiculous.

just in case nobody knows this, i get the following error when browsing to the CG siteadmin page:

Genesis :

Error creating new session

Back up for now.

By the way, according to the ads I pay for on Project Wonderful, I had 4000 page views yesterday.

Today?

About 100.

Man this hurts.

I'm going to coin a phrase here.

Internet Epilepsy.

May I just put in my two (or so) words to say,

Thanks for working at this. Now that I know it's a server issue, and that people are on it, I'm inclined to be more patient. As are most people, I'm sure.

And that being said, comickers, please keep in mind that ComicGen is a *free* service, and that if you don't want to have to deal with any issues that may come of their hosting (which you shouldn't complain about because it's free, and they always do the best they can), you can always find someone else to host your comic and start updating it yourself.

I've been here for a few years, and I think they're doing a great job.

illustar wrote: And that being said, comickers, please keep in mind that ComicGen is a *free* service, and that if you don't want to have to deal with any issues that may come of their hosting (which you shouldn't complain about because it's free, and they always do the best they can), you can always find someone else to host your comic and start updating it yourself.
I've been here for a few years, and I think they're doing a great job.

Having a big honking ad above our comic that doesn't give us a DIME is the price we pay. I don't know if it covers the cost of hosting everyone, but CG IS bringing in money. And although I complain when things go south, I think that's a pretty natural reaction.

Kisai mentioned that the Powers that Be won't supply a new server. So they are to blame for the server issues, at least in part. Donations and fund-raising is possible, as others have suggested.

That said, I am looking into setting up Lab Bratz on Webcomics Nation and DrunkDuck. The only things that have stopped me so far are the work of transferring over my archives and maintaining a presence on multiple sites. I just don't have that kind of time.

And I do like it here. I'm just really frustrated right now because of the frequency of the outages.

Ed

jameslong wrote:just in case nobody knows this, i get the following error when browsing to the CG siteadmin page:

Genesis :

Error creating new session

I'm getting the same thing.

And yeah, yeah. CG is free so therefore we're not allowed to complain or whatever, but let's be serious here. CG has offered a service and the service isn't being provided. We provide our end, the comics and other content, so it's reasonable to expect the service to actually work as advertised. And equally reasonable to be a wee bit frustrated when we work hard to produce comics and find it impossible to post them.

Thanks for taking the time to post an explanation, Kisai. I really appreciate it.

New thing going down now, attempting to log in give me this:

Genesis :

Error creating new session

No idea what that may mean to the programmers, but I do know that this means I can't update.

or for that matter check what position in the que you are for an update (wishful thinking I guess).

Bit of a shock to me to see that the manual update I triggered sometime friday morning PDT hadn't made it to my strip yet, but hey, you get that. Normally I notice things going wrong fairly quickly (with the exception of CGWiki, and can come here and find out whats going on within about the first day. But seeing as I work 12 hour shifts, and am currently rostered on, I'm basically living under a rock. Anyway, I'm way sidetracked from the whole point of this post ... which I can't remember anyway. Damn. Oh well, life goes on.

Good luck getting this fixed tech type people, I appreciate your efforts.