Days like yesterday are why.

Zero-Day Emergency

One of the benefits of hosting your site with a managed provider like Synthesis is that you have the peace of mind of knowing your site is on a healthy and secure web server. In most cases, we are able to provide such an environment with proactive software features like our brute-force attack blocker or our constant malware scanning.

However, sometimes the exercise of security turns into emergency patching that simply can’t wait. Yesterday was one of those days.

A Zero-Day exploit of the Linux kernel was made available that had potentially disastrous consequences for web hosts. This forced us to take immediate action in the form of an immediate patch and hard reboot.

We do not make a decision like a system-wide hard reboot lightly. It’s literally choosing immediate downtime for the sites we host. In this case, though, it was a no-brainer.

We will normally perform seamless upgrades, but the potential for a Zero-Day security disaster forced us to do the hard reboot without warning last night. Waiting 5-7 days — heck, even 5-7 hours –  was unacceptable.

So … this is why your site came down yesterday. It allowed us to apply a necessary patch. And now your site is safe.

As you know, we are content publishers ourselves. Our sites were impacted right along with yours. We hate downtime just as much as you do, but the reality of the moment was that we had no other choice but to react quickly.

So … thank you

I ended up personally fielding a number of the Site Down tickets that were submitted last night during the reboot, and I want to extend a quick note of thanks to all of you who wrote in. Your patience, understanding, and just generally great attitudes about the entire situation were much appreciated.

We know that you have many options when it comes to managed WordPress hosting, and we’re glad you choose us. I can’t imagine a more astute, enjoyable group of customers to work with than what we have at Synthesis.

Now back to your regularly scheduled peace of mind. We know it’s what you pay for, and it’s a responsibility we never cease working to fulfill.

***

Note: this post was published on May 15th, 2013.

This post will help you decide if action needs to be taken and what action you should take.

When to redirect

Most people’s first instinct upon seeing a 404 error is to consider a redirect. Hold on!

Redirecting any and all 404 errors can actually be bad for your search engine results. Via Google Webmaster Tools:

Returning a code other than 404 or 410 for a non-existent page (or redirecting users to another page, such as the homepage, instead of returning a 404) can be problematic. Firstly, a page like this (called a “soft” 404) tells search engines that there’s a real page at that URL. As a result, that URL may be crawled and its content indexed. Because of the time Googlebot spends on non-existent pages, your unique URLs may not be discovered as quickly or visited as frequently and your site’s crawl coverage may be impacted (also, you probably don’t want your site to rank well for the search query [File not found]).

The first thing to ask yourself is: did this content ever exist?

If it did, and you have changed your domain or URL structure, you will want to implement redirect(s) to send users and search engines to the right place.

If it never existed, don’t sweat it. As mentioned above “Redirecting any and all 404 errors can actually be bad for your search engine results.”

Troubleshooting

WordPress sites can complicate the 404 troubleshooting process. Why? WordPress is a content management system that processes its own internal rewrite array as a part of its permalinks feature.

The first step to troubleshooting 404s is to figure out whether the 404 is being caused by the web server or by WordPress.

Static file

If your file is static (e.g. a jpg image) open your FTP client and verify that the file exists.

As an example, let’s say that the URL http://mydomain.com/wp-content/uploads/picture.jpg is producing a 404 error. You will want to:

• Open FTP Client and connect to your server (if you don’t know how to do so, read this.)
• Navigate to the file’s location
• Verify that the file exists
• If it is does not exist, you have found the source of your 404.

However, it it does exist, yet pulling the URL up in a browser results in a 404 error, continue on to find out whether the issue is web server-based or WordPress-based.

In FTP create two files at the root  [/var/www/yourdomain.com/] for testing purposes.

First: Create an html file named test.html that has the words “It worked” in it. (Download this file here.)
After uploading it, head over to: http://yourdomain.com/test.html

If you see “It worked,” the server is processing static files correctly, so your issue is likely in WordPress.

Second: Create a PHP file named test.php (or download here) that includes the following:
<?php echo 'It works in PHP'; ?>

After uploading, head over to http://yourdomain.com/test.php. If it says “It works in PHP,” then PHP is functioning properly as well, so again your problem is likely in WordPress.

(Quick side note: make your mama proud by removing these files when you are done testing!)

Fixing 404s in WordPress

WordPress creates a rewrite array that is responsible for managing permalinks, or “pretty URLs.”

Improperly coded plugins, custom post types, and taxonomy alterations can break or corrupt the permalink array, thus causing 404s.

Quick fix: re-save permalinks

You can reset the permalink array by going to the permalinks tab in wp-admin and simply re-saving the permalink structure.

• Navigate to: WordPress Dashboard → Settings → Permalinks: Click “Save Changes”

Test your URL. If the 404 goes away, something caused an issue with your permalink structure and this fixed it.

Often the quick fix is enough to correct your 404 issues. However, should they come back, go on to the next troubleshooting step.

Find the cause

If the quick fix did not work or you are seeing 404s again, try disabling plugins that handle redirection or create customized permalinks. Did that solve it?

Next, use a debugger to view the rewrite array and confirm that the rewrite rule responsible for processing your URL is not in place. To do this, install the plugin Debug This, which makes it easy to view what is actually in the WordPress rewrite array.

Once this plugin is installed and activated, go to your site. Then navigate to:

• Homepage → Admin Bar → Debug This → Query → Rewrites.

You should end up on a screen that contains rewrite rules on the left hand side and the actual PHP string being rewritten on the right hand side. So what do you do with this information?

Let’s say we have a broken author feed (example: https://www.copyblogger.com/author/jerodmorris/feed/). To troubleshoot the broken feed, look for the word “author.” Eventually this rule is found: author/([^/]+)/feed/(feed|rdf|rss|rss2|atom)/?$

If you don’t find the rule, this is the source of your issue. WordPress will not process the request unless it knows what it is doing.

The actual PHP URL string that WordPress uses for this author feed is: http://copyblogger.com/index.php?author_name=jerodmorris&feed=rss. If this long PHP URL does not work, you have another problem with your site.

You will now want to start by deactivating plugins one by one to see if you can find the conflicting PHP responsible for the issue.

***

And, of course, if you go through all of these steps are still unable to fix the issue, submit a Help Desk ticket.

If you have to submit a ticket, take note: it is very important that you let us know the exact URLs that are giving unwanted or unexpected 404 errors, and that you walk us through the troubleshooting steps you have already completed. Then we can add some next-level troubleshooting and insight to get your problem solved.

These architecture decisions proved to be unpopular with some folks. In some cases, they actually cost us customers.

You might wonder, then, why we keep them in place.

Simple. Popular and right are not the same, often far from it, and we won’t compromise the security of everybody for the convenience of few.

That is why, for example, we do not allow phpexec() calls on our servers. Not even on a server-by-server basis. We don’t believe in it. It opens your site up to danger without enough of a system-wide reward to justify the risk.

And it’s why Synthesis-hosted sites were never in danger of being exploited by what Sucuri called a very serious vulnerability.

To recap: it was discovered recently that W3 Total Cache and WP Super Cache – two of the most widely downloaded WordPress plugins ever – possessed a remote code execution (RCE), vulnerability. What does this mean? As Tony Perez summed up on the Sucuri blog, “This means I can pass any commands I want to your server and they’ll execute.”

I’m sure you can parse the risk from that statement.

As soon as the Sucuri blog post went up, we received Help Desk tickets and tweets wondering how we were going to “handle the situation.” Our response today was the same as it was two weeks ago when panic erupted in the WordPress community regarding worldwide brute force attacks: nothing.

Well, nothing is perhaps the wrong word. Nothing new would be more appropriate. It’s just business as usual.

As soon as we realized the threat to W3 Total Cache, which we recommend as our preferred caching plugin, we tested the code injection on our own sites. It didn’t work.

Our refusal to allow phpexec() calls on our servers – unpopular as it may be with some folks – kept every site we host safe from the W3TC and WPSC vulnerability, whether the plugins were updated or not.

We take great pride in that. Synthesis customers should too. It’s why paying a premium for serious managed WordPress hosting is worth it.

How much are safety and peace of mind worth? We think a lot. That’s why for us the “cost” of this particular unpopular decision has always been worth paying.

*****

For the record: do update your cache plugin as soon as you’re able, but not out of fear … simply because it’s a best practice to always keep all plugins current.

The answer to this question becomes obvious once the true costs of building and managing a WordPress site are analyzed. And a current event in the WordPress world – one that is rather ominous at first glance – serves as a microcosm of the value that Synthesis customers enjoy.

It’s the value of peace of mind.

The global attack on WordPress sites

A large hosting company issued a blog post today warning its customers about an “on-going and highly-distributed, global attack on WordPress installations across virtually every web host in existence.”

And by “virtually every web host in existence” they mean Synthesis as well. And they’re right. There is very likely some nefarious hacking ne’er-do-well trying to brute force their way into your site as I type this and then again as you read it.

In fact, one of the biggest and most prestigious sites we host gets brute forced 500 times an hour even on days when WP sites aren’t global targets.

But here is the difference between the average hosting company and Synthesis: whereas the average hosting company is going through a bit of a fire drill right now (imploring customers to change passwords immediately, prepping them for possible downtime, etc.), at Synthesis today is just another day. It’s business as usual.

Over two years ago we noticed that brute forcing of wp-login.php was 1,000X that of standard services like FTP and SSH. We also noticed that the big name cloud-based security services our customers often put in front of us were doing nothing to stop it.

That was when we launched a well-crafted solution to guard our customer base against this threat. And we added wp-login.php brute force data analysis to our daily operational procedures at Synthesis.

Why Synthesis customers are not in panic mode

We’ll never discourage you from changing your password, of course. Heck, go ahead and do it every day if you want to. But do it out of a general concern for having a sound security strategy, not out of fear because of this brute force uprising.

Now to the larger point:

The purpose here is not to knock other hosting companies. Most do a fine job of serving their respective niche in the hosting communities. They provide good low-cost hosting to site owners of many different content management systems.

But for the serious WordPress site owner – the one whose business and revenue and livelihood rely on the uptime and security of their website – more specialization and preparation is necessary.

The reason we can tell you not to worry in the midst of Brute Force Outbreak 2013 is that we’ve been dealing with this for years and continue to pay close attention to it. We have systems in place to deal with this sort of attack, and they are not just turned on when the web excrement hits the fan (so to speak); they are always on.

Frankly, that’s what you should expect from a managed WordPress host. We don’t consider it an “extra” benefit you get from hosting with us. We consider it a minimum level of protection that we owe you for trusting us with your sites and paying us a premium to host you.

After all, we are content publishers ourselves and run our own sites on the exact plans and services make available to you.

So go on about your business this weekend. Watch the Masters. Go to yoga. Learn how to cha-cha. But don’t worry about the global brute force attacks hammering WordPress sites worldwide.

If you are the type that needs to be proactive, change your passwords to something strong. But again, don’t do it on account of the attack; do it because it is a best practice.

We’ve said it before and we’ll say it again: we’ve got your back.

And we mean it.

That’s why we are so excited to officially announce the latest Synthesis innovation: Synthesis Personal Backups for S3, a backup system that truly is unlike any other.

It is now live and available for all Synthesis customers to use.

And did we mention it’s free?

We’re serious about backups.

We already back up your site using Linux-level technologies that we’ve been evolving for the past 4 years.  These backups are sent to our own enterprise-class storage in a data center that sits almost 1,500 miles away from our servers.  And we monitor these for success on a daily basis.

Yet no matter how professionally managed this process is, you’ve told us that you won’t be comfortable until you can have copies of your site’s backups sent to you.

You want your site backed up and in your possession. We agree with you!

So we recently released Database Snapshots as part of our Software Monitor must-use plugin. This allows you to easily backup your database before making WordPress updates.

Synthesis Personal Backups for S3 is the next step in this continued volution.

Synthesis Personal Backups for S3

This system allows you to configure files, directories, and databases that you want backed up.  Our robust server processes then push those files to your S3 account.

Unlike PHP-based backup plugins that have significant limitations, our processes are driven at the Linux level and use both file transfer and file synchronization technologies made available by Amazon.

We are a little bit different in our approach to backups. We are more concerned about fast and effective recovery than creating huge, unmanageable backups.  It is not just about the ‘activity’ of backing up; we view the end game as successful recovery from site issues.

Why restore your entire site and lose comments when all you need is a CSS file from 5 days ago?

Thus, we’ve created a user guide (available to Synthesis customers in your site’s dashboard via the Software Monitor plugin) that not only walks you through configuration but also provides you with disaster recovery strategy.

We’re content publishers too, and we live with the need for these solutions.  That’s why we develop them.

Amazon’s Simple Storage Service (S3) is reliable and affordable.  There is no reason not to maintain a second set of backups for your site.

If you already a Synthesis customer, get started by logging in to your wp-admin, clicking on ‘Software Monitor’ (underneath the Dashboard tab), and then clicking on the user guide link in the Personal Backups for S3 section.

If you need some input, just ask us.

Answers to a few FAQs

Q:  I have a backup plugin I am currently using.  Why should I disable it and use Synthesis Personal Backups for S3?
A:  Backup Plugins are written in PHP and do fail. Moreover, they tend to backup your site locally on your hosting account, which makes no sense at all because IF you lose all the data in your site … you lose your backups as well.

You can use what you’d like, but we use Synthesis Personal Backups for S3 for our own sites and we encourage you to try it out.

Q:  How is Personal Backups for S3 different from Database Snapshots?
A:  Database Snapshots make a single copy of your database. This tool is designed to give you a known good copy of your database prior to making updates to plugins, themes, or WordPress that change the database.

Personal Backups for S3, on the other hand, is designed for disaster recovery. It allows you to backup files, databases, non-WordPress databases, and retain these for defined periods of time.

Moreover, Personal Backups for S3 has a synchronization component allowing you to synchronize parts or your entire site to S3 for maintaining a “hot copy” of the files.

Q:  Why does the Database Snapshots UI not have a restore button?
A:  We are in talks about that now. The technology to do this is easy. Still, we’d hate for you to lose post or comment data when all you need is to restore wp_options.

If you need to restore a DB Snapshot, open a support ticket for now and our Help Desk staff will walk you through the right course of action. Soon we hope to release a “Smart Restore” function to that user interface.

Q: What is Synthesis Plugin Snapshot Reports?
A:  Synthesis Plugin Snapshot Reports allow you to create up to 10 reports of the versions of plugins, both active and inactive, on your site at any given time. We highly encourage the removal of inactive plugins for security reasons and these reports allow you to reference versions of plugins you’ve had on your site at a given point in time.

We do not backup the plugins when you take the snapshot. It is just a picture of your site’s plugins and their versions at a moment time. So to bring a deleted plugin back, you would need to go to WordPress.org to download previous version of a plugin.

Q:  When will you support other cloud storage like DropBox or Azure?
A:  This is not in our plans. Storage is a commodity, and we chose Amazon primarily due to the quality of their Linux libraries as well as for their reliability and affordability.

An average Personal Backups for S3 configuration will cost less than $5 a month at Amazon.

The upgrade is easy. You should take this opportunity to get your site up to date with the latest version of Genesis.

For Synthesis customers wanting to complete this upgrade, it is as simple as clicking the update link in the message bar on the back end of your site. However, first you should take advantage of our Database Snapshot utility (located inside the Software Monitor) to make a quick backup of your database. This is a good standard operational practice.

And if your site is still running a version of Genesis prior to 1.9, you really should take this opportunity to update your site, as we’ve fixed bugs and even a few potential security holes since the initial 1.8 release.

Bottom line:

• WordPress.org has updated the plugin to remove the malicious code.
• We immediately initiated a system-wide update of the plugin on all Synthesis sites using it. This update is complete.
• The plugin is being removed from the repository and we strongly advise you to find an alternative plugin as soon as possible.

Find an alternative

We debated simply removing the plugin immediately from all sites using it, but that could have compromised the functionality of sites using it. We didn’t want to do that.

But do not be lulled into a false sense of security by the safe version 4.0.1. It’s just there as a placeholder, created by the WordPress team so the malicious code could be removed without immediate impact to functionality. But it won’t be updated long-term.

So it is safe to use for now, but you need to find an alternative. Unsupported plugins are not good for your long-term security.

Note: at this time we do not have an official recommendation for an alternative. Feel free to comment below if you know of or find a good one and we’ll vet it.

More detail

If you would like more details about the breach, read about it on Sucuri’s blog here. They were all over this last night, which is why we continue to value their partnership in keeping your websites secure.

The WordPress team was also all over this. They realized the breach quickly, removed the plugin from the repository, submitted the stable update in its place to prevent its users from having to forgo its functionality, and they will most likely never allow the plugin back in the repository.

Files like your theme’s functions.php, if edited incorrectly, even if one little semicolon is missing or out of place, can whitescreen the front end and back end of your site. And there is no Edit — Undo function inside of WordPress to quickly rescue you from trouble.

If can’t log in to access the WordPress editor, how are you going to fix it? Answer: the way you should have been editing in the first place.

FTP.

You have probably heard the term a lot, but you may not know what it means. FTP stands for File Transfer Protocol. You can find out much more about its history here.

More importantly, you might not know how it works. Let’s fix that.

In this article we will be using the Cyberduck FTP client. It is open source and has versions for both Mac and Windows. (It’s free, so if you try it and keep using it make sure to donate.)

• Get it here: cyberduck.ch
• Follow the installation instructions to get it working on your computer.
• Come on back.

Getting Connected with FTP

The most difficult part of using FTP is filling out the credentials correctly to get connected. Thankfully, the bookmarks tool in Cyberduck means you will only have to do this once.

Make sure you have your FTP credentials handy. If you are a Synthesis customer, your FTP credentials are available in your Customer Portal.

The following screen shots and instructions will take you step-by-step through the connection process:

Step 1: Click “Open Connection”

Step 2: Click the Dropdown

Step 3. Select Your Protocol

This can differ depending on your host. If you have trouble connecting with one, try another.

Synthesis Customers should choose SFTP (SSH File Transfer Protocol), as indicated in the Customer Portal.

Step 4: Add Your Credentials

Enter in the fields for Server, Username, and Password.

Step 5: Click on “Connect”

Once your credentials have been entered, click connect.

Step 6: Bookmark it

Bookmark it now before you forget, that way next time all of this can be accomplished with one click!

In the top menu:

• Bookmark –> New Bookmark

Important FTP Ground Rules

Now that you have installed Cyberduck we need to lay some ground rules before going any further.

1. Save a copy of the file to your computer before editing.

There is no undo or trash in FTP. Once you delete or replace something it is gone … unless you saved a copy. Always be safe rather than sorry.

2. Use a text editor.

To edit document files you will want to use a text editor, some of our favorites are:

• Mac – TextWrangler: http://barebones.com/products/textwrangler/
• Windows – Notepad++: http://notepad-plus-plus.org
• Windows – Notepad2: http://flos-freeware.ch/notepad2.html

And we’re in the final testing stages of making it a part of our offering at Synthesis.

What is SPDY? Put simply, SPDY is an application-layer protocol that reduces latency. It does this by maximizing the strengths of HTTP and TCP, the current standards, while making up for their weaknesses in a number of ways. (For example, SPDY enables concurrent HTTP requests to run across a single TCP session.)

If you want learn the nitty-gritty technical details, click here.

If you want to know in layman’s terms how it can impact you and your site, just know that it can reduce load times by as much as 60%. That’s why sites like Facebook and so many others are rushing to become SPDY-fied.

But there’s a catch. Both the server and the web browser have to be SPDY-enabled. Fortunately, a growing number of web browsers are becoming SPDY-enabled. Forward-thinking web hosts are following suit.

Here at Synthesis, we want you to have every tool available to make your site faster. Because Synthesis sites run on NGINX, one of the earliest SPDY supporters, we can do just that.

And since we always eat our own dog food, we’ve gone to work testing our SPDY configurations on our own sites. One of our biggest, My.StudioPress.com, is SPDY-enabled right now.

Why My.StudioPress.com and not the entire site? Because SPDY is right for some sites and not right for others.

We are in the process of testing a number of other sites, including select customers who trust us with beta configurations. After this testing is complete, full SPDY support will be available for qualified customers.

Follow us on Twitter or subscribe below to stay up to do on this and other Synthesis developments.

First and foremost, upgrades and updates should be done during your off-peak hours, i.e. your low traffic time. Usually this is at night (when people are not bored at work surfing the web), but it depends on your particular site’s traffic patterns.

The key here is to execute your update during a low activity time just in case something goes awry.

Next, start working through the steps below.

Depending on what you need to update, some of these steps may need to be omitted, but here is the proper sequence of update events:

1: Back Your Site Up

Backups ensure that you have an immediate action plan should something go wrong (other than flailing your hands in the air while screaming expletives).

• Back up the database of your site. You can do this either via PHPMyAdmin or with our new dashboard tool, Software Monitor, which you will find in the sidebar menu just beneath “Updates.”
• Back up your themes and plugins. (Synthesis does this for you nightly, but you can take manual backups via FTP if you want to.)

It is worth noting that backing up the uploads folder (inside wp-content), which can often take a long time, is not necessary before upgrading. We have backups of it on hand, plus nothing in a WordPress update is going to affect your uploads.

2: Turn Caching Off

Deactivate your caching plugin. In most cases this will be W3 Total Cache, Super Cache, or Quick Cache.

This ensures that you aren’t caching maintenance pages during the updates and are not otherwise interfering with the update process.

3: Update Your Plugins and Theme Framework

Often, WordPress updates are accompanied by plugin and theme updates. Developers usually have a head’s up on impending WordPress updates and can thus ready their plugins to be compatible when updates are released.

Update all of your plugins and themes before upgrading WordPress.

4: Update WordPress

Now it is time to update WordPress itself! Just click the automatic upgrade button and let WordPress do its thing.

Sometimes it can take a handful of seconds to complete, so don’t get impatient if you see a momentary blank section within the dashboard.

5: Reactivate Your Caching Plugin

Reactivate your caching plugin and clear out/purge the cache.

If you are using Synthesis and W3 Total Cache and don’t have a customized configuration, now is a great time to update your W3TC settings to our preferred settings (which you can download here).

6: Check Your Site

Take a minute to look over your site and visit a few pages and posts. If something isn’t working properly, it is best to catch it now instead of later.