Im considering taking web server from China to reduce site loading times from China/China users. Problem is, how to sync/keep same data between two sites? When editing content in the site it should update these changes to site in China server.
Server is running Linux, Apache and MySQL. Website is using WordPress.
FYI I'm already using CDN and site loading speed is still too long from China.
Basically your solution would need to...
Copy the entire contents of your http'd directory from the main server to the Chinese server.
Copy the entire contents of your MySQL database from the main server to the Chinese server.
Perform these tasks at a regular interval without manual intervention.
I can guide you to references that will help with each task and sometimes can show you a quick example. However, if you want to get it to work and especially if you want to optimize the process, you're going to have to look through the references yourself.
If I didn't do it this way this answer would get even more horrendously long that it already is.
Before we start you should remember...
Thing 0 - Please Try Not to be Intimidated by the Length of this Answer
I know I've written a lot, perhaps more than I should have, but I guarantee you are capable of implementing this in no more than a day. I have tried to be thorough but that does not mean that what I'm describing is particularly complicated.
Thing 1 - Shutdown your Chinese Server During Transfer
This transfer of data is going to make your Chinese server unusable while it's in progress, as you might have guessed. You need to make sure that you're Chinese server is not operational during the transfer. Otherwise the server might have only partial data available which could cause problems for both client and server, particularly in relation to MySQL.
Thing 2 - Use Compression as much as You Can
As time consuming as compression and decompression can be for large amounts of data, believe me it is nothing compared to the time you will waste sending the uncompressed data to China. Network usage, not processor time, is really going to be the limiting factor in getting the transfer done quickly. Try to send compressed files whenever possible.
Thing 3 - Try to Use Checksums
Sending all your data, particularly in compressed format, will leave it vulnerable to corruption in transit. Whenever you send a file I encourage you to use some kind of checksum on the data to verify that it has not been corrupted. For brevity I will not be showing you how to do this but I'm sure you're smart enough to figure out how to pepper in some verification.
In case you're not familiar with checksums, the Wikipedia article about them is pretty straight forward. The most commonly used are the MD5 and the SHA-1, but both of those are somewhat collision prone. I would recommend the SHA-2 (also called SHA-256/512) or the very new SHA-3.
Copying your Http'd Directory to the Chinese Server
As far as I know (and I could be wrong) there is no built in way to transfer files from one Apache server to another...so you're going to have to write your own script for this.
You're also going to need to have two separate scripts: one for the main server and one for the Chinese server. Here's a breakdown of what each script needs to do.
On your main server...
Log in as you're Apache server's user. (Reference for switching users.)
zip/gzip/tar.gz your http'd directory's contents. (Reference for zip. Reference for gzip. Reference for tar.)
scp (secure copy) the compressed file to your Chinese server. Make sure to copy it to the username that Apache runs under. (Reference for scp.)
Delete the compressed file.
Initiate the Chinese server's script (this will be discussed later).
You will likely be using a shell script for all of this, so I hope you're familiar with the terminal. A simple example would look like this.
#!/bin/sh
## First I'll define some variables to explain this better.
APACHE_USER="whatever your Apache server's username is (usually it's www-data)";
WWW_DIR="your http'd directory relative to ~ (usually it's /var/www)";
CHINA_HOST="the host name/IP address of your Chinese server"
CHINA_USER="Apache's username on the Chinese server";
CHINA_PWD="Apache's user password on the Chinese server";
CHINA_HOME="the home directory of the Apache user on your Chinese server";
## Now to the real scripting. I will be using zip for compression.
su - "$APACHE_USER";
zip -r copy.zip "$WWW_DIR";
scp copy.zip "$CHINA_USER#$CHINA_HOST:$CHINA_HOME" < echo $CHINA_PWD;
rm copy.zip;
## Then you initiate the next step of the process.
## Like I said this will be covered later.
On your Chinese server...
Log in as the Apache user.
Delete the content of the http'd directory (probably /var/www relative to ~).
Decompress the scp'd file (this will change depending on how you compressed it).
Copy the decompressed directory to the http'd directory (this step is unnecessary if you choose to compress with zip).
Deleted the compressed, scp'd file.
Notify main server to continue next step (again, will be discussed later).
This is pretty straight forward and I don't think you need another example for this part.
Copying the MySQL Database Contents
You can find a good reference for how to do this in this article from the MySQL website. Basically copying database contents is a built in feature. Try to make use of the compression options!
Performing these Tasks at Regular Intervals without Manual Intervention
Ok this is where things get kind of complicated.
The first thing you need to know is how to schedule tasks at regular intervals on Linux. This is done with a command line tool called crontab. You can see good examples for setting up cron jobs in this article, and the full crontab documentation here.
However what will take more skill than just scheduling the job at regular intervals will be synchronizing the data transfer. If you simply set one server to send data at a certain time and the other to receive it at a certain time, you will get many bugs. Be sure of that.
My recommendation would be to create a socket in the Chinese server that listens for instructions from the main server.
This can be done in a variety of languages. Because you're using Linux I would recommend doing this in C, but it can be done in almost any language including Bash.
A full example would be too much but basically this will be the flow of what you have to do.
Socket in China listens for connections.
Cron job in main server connects to China socket.
Main server authenticates itself.
Chinese server stops Apache, stops accepting requests.
Chinese server acknowledges authentication approved.
Main server scp's website contents to Chinese server.
Main server tells Chinese server that scp is complete.
Chinese server replaces Apache's http'd directory's contents with the data that has been scp'd.
Chinese server announces success to main server.
Main server copies MySQL data.
Main server tells Chinese server process is complete.
Chinese server resumes Apache service.
Chinese server notify's main server that service is resumed.
Socket is closed.
Chinese server goes back to listening for connection from main server.
I hope this helps!
Related
This is mainly curiousity. I may implement it, not sure.
I had this idea... Rather than leave a script on a server, per chance a hacker got to it, I could send the full script to the server from a secondary site, run it, and then delete it. OK in theory ...
The LWP::UserAgent on the 'sending' server would need to connect to a script on the 'receiving' server. The way I would do it at present would be to save the incoming param to a file using open filehandle etc, then a "require" to run the script, followed by an unlink to destroy the file.
$d=param('d');open (D, ">sc.pl");print D $d;close(D);require "sc.pl";unlink "sc.pl";
But a hacker could put a suitable "print" between most of those entries, and see what data was being sent.
So I wondered if a similar thing could be achieved WITHOUT writing the code to a file, ie to be executed whilst in memory. It might have something to do with a "while(<>){" or a pipe from STDIN to ???
Just a thought
The goal here seems to not be to secure the server, but to protect your source code from being stolen. That's the first flawed premise: hackers want your data, not your code. Hackers rarely care about your source code except to find security holes (which, if they control your server, they've already found). They're not stealing your code for industrial espionage, they want your data to either sell or hold ransom. Credit cards. Customer information. Passwords. Encryption keys. Any of that.
Second, if this theoretical hacker can alter the server code to add print statements then executing the program from memory or STDIN won't help that. If they control the server code they can print whatever they want no matter where the server reads it from. If they can edit the code, they can replace the entire server program.
This idea is a non-starter. You need to rethink your premises.
Current implementation-
Divide the original file into files equal to the number of servers.
Ensure each server picks one file for processing.
Each server splits the file into 90 buckets.
Use ForkManager to fork 90 processes, each operating on a bucket.
The child processes will make the API calls.
Merge the output of child processes.
Merge the output of each server.
Stats-
The size of the content downloaded using the API call is 40KB.
On 2 servers, the above process for a 225k user file runs in 15 minutes. My aim is to finish a 10 million file in 30 minutes. (Hope this doesn't sound absurd!)
I contemplated using BerkeleyDB but, couldn't find how do I convert the BerkeleyDB file into normal ASCII file.
This sounds like a one-time operation to me. Although I don't understand the 30 minute limit, I have a few suggestions I know from experience.
First of all, as I said in my comment, your bottleneck will not be reading the data from your files. It will also not be writing the results back to a harddrive. The bottleneck will be in the transfer between your machines and the remote machines. Your setup sounds sophisticated, but that might not help you in this situation.
If you are hitting a webservice, someone is running that service. There are servers that can only handle a certain ammount of load. I have brought down the dev environment servers of a big logistics company with a very small load test I ran at night. Often, these things are equipped for long-term load, but not short, heavy load.
Since IT is all about talking to each other through various protocols, like web services or other APIs, you should also consider just talking to the people who run this service. If you have a business-relationship, that is easy. If not, try to find a way to reach them and to ask if their service is able to handle so many requests at all. You could end up with them excluding you permanently because to their admins it looks like you tried to DDOS them.
I'd ask them if you could send them the files (or an excerpt of the data, cut down to what is relevant for processing) so they can do the operations in batch on their side. That way, you remove the load for processing everything as web requests, and the time it takes to do these requests.
I've read that things can go wrong with your web server which may lead to display of PHP scripts as plain text files in a web browser; consequently I've moved most of my PHP scripts to a directory outside the web root. Now I've been wondering whether the same could happen to the CGI scripts in my cgi-bin.
My main concern is one script which contains a user name and password for my MySQL database. If this is a possible security hole (at least as far as the database content is concerned), is there a way of putting sensitive data in a different location and getting it from there (like saving it in a file in a different directory and reading it from that file, for example)? My scripts are written in Perl btw.
I've read that things can go wrong with your web server which may lead to display of PHP scripts as plain text files in a web browser; consequently I've moved most of my PHP scripts to a directory outside the web root. Now I've been wondering whether the same could happen to the CGI scripts in my cgi-bin.
Yes. If something goes wrong that causes the programs to be served instead of executed, then any of their content will be exposed. It is exactly the same issue as with PHP (except that given the way that cgi-bin directories are usually configured (i.e. aliased to a directory outside the web root), it is slightly harder for the problems to occur).
My main concern is one script which contains a user name and password for my MySQL database. If this is a possible security hole (at least as far as the database content is concerned), is there a way of putting sensitive data in a different location and getting it from there (like saving it in a file in a different directory and reading it from that file, for example)?
Yes. Exactly that, just make sure the directory is outside the webroot.
For additional security, make sure the database only accepts the credentials for connections from the minimum set of hosts that need to access it. e.g. if the database is on the same server as the web server, then only let the credentials work for localhost. Causing the database to only listen on the localhost network interface would also be a good idea in that case.
My scripts are written in Perl btw.
I'd look at using one of the Config::* modules for this.
One concern worth mentioning is specific to shared hosting.
If you're on a host shared with other users, it may be impossible to hide the password from them.
This depends on configuration details for the OS and the webserver.
For instance, it is common to have an Apache configuration on Linux on which the only way for a user offering a website to make files readable or writable to the webserver user is to make them readable/writable to all users.
You may trust all of these users not to abuse this themselves, but if one of these websites has a vulnerability that allows intruders to view the full file system, the intruder can then exploit that on all other websites.
There are countermeasures against this, but they complicate things for the users, so many hosters don't implement them.
It's definitely not a good idea to hardcode a password in a script if you can avoid it. Fortunately both Postgres and MySQL support loading DB credentials from a file. For Postgres you use ~/.pgpass and for MySQL I believe it's ~/.my.cnf. In either case you would adjust the permissions so that only the user running the script has permission to read the file. The advantage of this approach is that you don't have to write the code to read the file - the DB client library does it automatically.
It is definitely a security concern. You should store the password encrypted in a separate file and make sure that only your app has access to it.
If you use directory configured as cgi-bin, there is no way for file to be shown except error with Apache configuration. If you use Perl programs outside cgi-bin directories but inside site root, it may happen.
Also, you may configure DB to accept connections only from local socket, so knowing DB password would be useless.
You've already gotten better answers than I can provide, but as a note:
It's very bad form to store passwords as plaintext, period.
In the same way it's very bad form to overwrite or delete files without asking permission. If you do it, it will bite you or your client in the butt eventually.
The application is simple, an HTML form that posts to a Perl script. The problem is we sometimes have our customers upload very large files (gt 500mb) and their internet connections can be unreliable at times.
Is there any way to resume a failed transfer like in WinSCP or is this something that can't be done without support for it in the client?
AFAIK, it must be supported by the client. Basically, the client and the server need to negotiate which parts of the file (likely defined as parts in "multipart/form-data" POST) have already been uploaded, and then the server code needs to be able to merge newly uploaded data with existing one.
The best solution is to have custom uploader code, usually implemented in Java though I think this may be possible in Flash as well. You might be even able to do this via JavaScript - see 2 sections with examples below
Here's an example of how Google did it with YouTube: http://code.google.com/apis/youtube/2.0/developers_guide_protocol_resumable_uploads.html
It uses "308 Resume Incomplete" HTTP response which sends range: bytes=0-408 header from the server to indicate what was already uploaded.
For additional ideas on the topic:
http://code.google.com/p/gears/wiki/ResumableHttpRequestsProposal
Someone implemented this using Google Gears on calient side and PHP on server side (the latter you can easily port to Perl)
http://michaelshadle.com/2008/11/26/updates-on-the-http-file-upload-front/
http://michaelshadle.com/2008/12/03/updates-on-the-http-file-upload-front-part-2/
It's a shame that your clients can't use ftp uploading, since this already includes abilities like that. There is also "chunked transfer encoding" in HTTP. I don't know what Perl modules might support it already.
I have my server maintaining the content with a file-system(i mean folder structure). The same folder structure is also maintained in my iPhone client application bundle too.
Now if there is a change in my server file system(Add,Delete,Update of a file in some folder in the hierarchy) i need to update the file system accordingly at the client. This means that i need a protocol to be followed b/w the server and the client.
Can any one suggest how can this be done?
--
Thanks and Regards,
U'suf
From what I can tell, there is no easy way. I was looking for an rsync equivalent, but I haven't found one.
In my case, I'm manually walking the tree asking the server for differences after a certain date and I remember the last successful sync date.
Not pretty. Could spend lots of time coming up with something sophisticated.