Nodes map
Let's begin from the worker servers: As it's shown in the pic, I have two
Raspberry's with 512GB ram working on 700Mhz
as servers storing my website. All of source code, images, icon files and others are storaed by them. As an extra, 1st rasPi has a wifi dongle plugged on it. these two also hawe scripts on them, which do following jobs: when I scp web sites updated version in .tar.gz format, it extract's it and sync with the second one. However, my concerns are about the load balancer. Load balancer stores database that website will be using all the time and Hard disk drive(cloud storage that user will be reaching via veb site as well). As I mentioned, this raspi also possesses the job of load balancing. Will it overload due to this much task? If it will, is there a solution for that? Thanks in advance! All answers will be appreciated...
Related
So I am interested in server PCs and I want to buy one, and I will choose very powerful. But I don't know how to establish the hard disk to be connected to the internet. I want other people to see it when they write it's domain in the search. I am just searching for advice.
I went back over your question and this thread and this is what I recommend. You are looking to create a hosting environment for others from what I am understanding. Regardless the platform you select (linux or windows) having a beefy machine is going to be key to this. I would recommend at a minimum for hardware the following specifications. I recommend building a dedicated server with multiple Quad Core processors, 32 GB RAM, 2 or more TB Disk, with provision for backups. If you call say Dell or one of the other big server providers, they can custom-create a build for you that will accommodate your needs. That configuration would be a start; your final build may be beefier according to your needs and budget.
I have a social networking site which is almost ready. On the site people would upload images and put information about themselves for their profile and would also post messages (which can include images). I am wondering exactly how to proceed (hosting, servers etc.), I am a relative beginner at all this stuff so I am not sure exactly what route to take. I am thinking of maybe hosting from home initially from my Personal Computer and maybe expand by acquiring servers to stack (which I am not exactly sure how to do honestly) if we grow. Since the site is aimed at a small proportion of the population, I am not expecting huge growth in traffic but I want to be prepared for spikes, albeit small ones. I was wondering if maybe it is possible to just host it off my computer and store the the database (MySQL) in a removable disk along with the images. I was also thinking about cloud hosting, which seems to be the most common, but I was wondering if that really is the best thing to do, given this is a social networking site. I know this question is very vague and broad, but since I am a beginner I really have no clue how to proceed. What is the best thing to do? Thank you so much!
Hosting from personal computers is a bad idea for few reasons - your internet bandwidth limits the speed of the website, you need to maintain 24/7 interest connectivity/ power and all the resources.
I suggest you to start with AWS, get a free account of AWS, which comes with a basic level machine free for 12 months, more details here (https://aws.amazon.com/activate/).
Deploy a machine in EC2,
Install the webserver and MySQL tools into the machine
Host your files in this machine.
Refer this machine public ip to your domain service provider(where you bought your domain. Example: Godaddy)
Deploying a machine and configuring the server takes a while, but its worth, it and the best part is its FREE for 12 months, so you need not worry about the pricing, connectivity and bandwidth.
Also when you think the traffic is more, you can upgrade your server with few clicks with no config changes.
This is rather a set of questions than one very specific question. In the last couple weeks/days I puzzled together information regarding how to properly host a JAVA PLAY application "in the cloud", as lots of this information is scattered over different services, I felt like gathering up all these small pieces to one, because lots of things are important to be seen in full context. However, I moved my considerations to the bottom of the question, as they are mainly my opinions and subjective findings, which I don't want to be held responsible for. If I got something wrong, please don't hesitate to point that out.
Hosting Java PLAY + MySQL on AWS for world wide accessibility
Our Scenario: we have a quite straight forward application written within the Java PLAY framework (https://www.playframework.com/), working on iOS and Android as well as with a backend-system (for administration, content management and API), storing data in a MySQL DB. While most of the users' interactions with the server is quick and easy (login, sync some data) there are also some more data-intensive tasks (download some <100mb data zips to the mobile phone, upload a couple of mb to the server). Therefore we were looking for a solution to properly provide users far away from our servers with reasonable response times. The obvious next step was hosting in the cloud.
Hosting setup within AWS:
Horizontal scaling: for the start, only 1 EC2 instance with our app will be running in eu-1a. We will need to evaluate how much resources one instance actually requires, if more instances are needed and if more instances would actually benefit to quicker response times.
Horizontal scaling across regions: once the app generates heavy user load from another region, the whole EC2 instance should be duplicated and put to another region, running a db read replica (see Setting up a globally available web app on amazon web services and https://aws.amazon.com/de/blogs/aws/cross-region-read-replicas-for-amazon-rds-for-mysql/ ).
Vertical scaling of EC2 instances: in recent tests of the old hosting setup, the database proved to be the bottleneck rather than the play app and its server's hardware specifications. Therefore it is not yet fully clear how much vertical scaling would affect response times. If a t2.micro instance serves as good as a m3.xlarge instance, of course we would rather climb our way up from the bottom here.
Vertical scaling of RDS: we will need to estimate how much traffic hits the DB server and what CPU/RAM/etc will be required. Probably we will work our way up here aswell.
Global Redirection: done using Amazon Route 53 (?). A user from Tokio should be redirected to the EC2 instance running in Asia; a user from Rome to the EC2 instance in Europe. This does not only affect API calls within the app, but also content delivery (in both directions).
Open Questions regarding the setup
Is this setup conclusive? Am I missing crucial components?
Regarding global redirection: is Amazon Route 53 the right tool? How does it differ from CloudFront (which strikes me to be purely for content / media distribution?).
How do I define correct data/api endpoints for my app? Of course I don't want to define the database endpoint of a db read replica during app deployment. Will this also happen during the AR53 (question 2) setup? Same goes for API calls, of course the app should direct it's calls to https://myurl.com/api and from there it should be redirected. Is this realistic?
I would highly appreciate all kinds of thoughts (!), also regarding the background info written below. If you can point me to further reading to solve my questions on my own, I am also very thankful - there is simply a huge load of information regarding this, but this makes it hard to narrow the answers down. I do have knowledge in hosting/servers, but I am pretty sure there are true experts out there waiting to slap me with knowledge. :)
Background-Information
Current Hosting Setup: a load balancer distributes the traffic on 2 root linux servers, both of them running the PLAY app, one of them also holding the MySQL installation.
The current hosting setup has 3 big flaws:
No vertical scalability: the hosting company would take money for each scaling step. Currently the servers are running idle, but if the app booms, we could run short on capacity quickly. Running idle is still paid as if permanently under full load. This is expensive!
No deployment support: currently, we connect through SSH, manually deploy the correct folders to the file system, recompile on the server, set privileges, apply database evolutions; do the same for the second server (with different db connection parameters). What could possibly go wrong. ;)
No worldwide availability: to set up another server in another region of the world would mean a huge effort. To have a synchronized replica of our DB can be done, but once again deploying would mean downtime, room for errors and therefore time and money.
Hosting Options for Java PLAY:
There are lot of different blog posts about this. In short:
AWS: Amazon Web Services is one of the first places you start looking. Here you get everything that's possible, at a flexible price. You set yourself up an EC2 instance, a MySQL RDS and you're good to go - all of this in the free tier, so you can experiment, play around, test your stuff.
Microsoft Azure: similar to AWS regarding pricing and possibilities. However, I did not dive into setting up and deploying our application for test purposes.
Heroku: super easy deployment from within PLAY, scalable servers. However (on the first glance?) lacks possibility to supply remote regions with high speed content.
Jelastic: even easier deployment from within PLAY / IntelliJ IDEA. You push your app image to jelastic, jelastic distributes it further to their infrastructure providers.
RedHat OpenShift (https://www.openshift.com/): sounds promising, yet not as complete as AWS.
Lots of choices and possible setups/prices. Especially after finding out about deployment using boxfuse (https://cloudcaptain.sh/) I made my choice for AWS, as it offers absolutely all we need from 1 source. Boxfuse has low monthly costs but is perfectly integrated into AWS. Scaling is supported as well as the 3 common environments (dev/test/prod). Support is outstanding.
The setup looks good. I would however make one change: your large up- & downloads. As mobile speeds may not be ideal, have your app serve long-running requests is something you should avoid as this will needlessly tie up server threads. Instead consider having users upload and download straight from S3 using presigned URLs. You can then later add CloudFront to the mix when it makes financial sense to do so.
R53 will work just fine for picking the best server(s) for each end user.
For EC2 consider having an ELB + Auto-Scaling Group setup. Even just for a single instance you get the benefit of permanent health monitoring and auto-respawns. If you expect more load you can then auto-scale based on your expected bottleneck (cpu, network i/o). This will give you a more autonomous and robust setup than manually having to scale up and down based on your own monitoring analysis (even though the scaling part is very easy if you stick with immutable infrastructure & blue/green deployments like what Boxfuse offers).
Your focus on vertical server scaling might not serve you well on AWS. I would start thinking about horizontal scaling of app servers behind an Elastic Load Balancer, and possibly look into Elastic Beanstalk.
I'm not sure you can setup a read replica in another region via RDS, you might have to set that up via MySQL servers running on standard EC2 instances. And even if you can, that's going to be some expensive and high-latency data transfer.
If file uploads and downloads are all you are worried about, you just need to put CloudFront (Amazon's CDN service) in front of your application, and allow it to handle file uploads and downloads via its global edge servers. You could even do this without moving your entire application into AWS. I would recommend reading this blog post as a start.
I want to have two tcp connection in a single machine via socket programming, But this two connection should connect to two different network interfaces. One is say my 3g dongle and the other is wifi modem. But is it possible for a single machine(OS) to have two connection active at a time? If possible how to program the tcp connection via socket programming?
This can definitely be done, if you just create two programs and run each of them, they will both be able to communicate over their respective network. When you run a program, the operating system creates a process dedicated to running that program, which is assigned time on the CPU by the scheduling algorithm in the OS. So long as your CPU can keep up with any processing associated with the networks, they will both be able to run simultaneously.
You make no mention of your plans for this, but be aware that I/O times can also limit your speeds. If you're using an older computer, it may not be able to transmit a lot of data very quickly due to an out-dated (or just low powered) network card.
Next time try to research your question first, information about this can be found with relative ease using any popular search engine, including the search bar at the top of this page. Also read this, or one of the other several help articles about asking questions well, that are available from the page you had to go through before asking the question.
I'm looking for ways to gather files from clients. These clients have our software and we are currently using FTP for gathering files from them. The files are collected from the client's database, encrypted and uploaded via FTP to our FTP server. The process is fraught with frustration and obstacles. The software is frequently blocked by common firewalls and often runs into difficulties with VPNs and NAT (switching to Passive instead of Active helps usually).
My question is, what other ideas do people have for getting files programmatically from clients in a reliable manner. Most of the files they are submitting are < 1 MB in size. However, one of them ranges up to 25 MB in size.
I'd considered HTTP POST, however, I'm concerned that a 25 mb file would often fail over a post (the web server timing out before the file could completely be uploaded).
Thoughts?
AndrewG
EDIT: We can use any common web technology. We're using a shared host, which may make central configuration changes difficult to make. I'm familiar with PHP from a common usage perspective... but not from a setup perspective (written lots of code, but not gotten into anything too heavy duty). Ruby on Rails is also possible... but I would be starting from scratch. Ideally... I'm looking for a "web" way of doing it as I'd like to eventually be ready to transition from installed code.
Research scp and rsync.
One option is to have something running in the browser which will break the upload into chunks which would hopefully make it more reliable. A control which does this would also give some feedback to the user as the upload progressed which you wouldn't get with a simple HTTP POST.
A quick Google found this free Java Applet which does just that. There will be lots of other free and pay for options that do the same thing
You probably mean a HTTP PUT. That should work like a charm. If you have a decent web server. But as far as I know it is not restartable.
FTP is the right choice (passive mode to get through the firewalls). Use an FTP server that supports Restartable transfers if you often face VPN connection breakdowns (Hotel networks are soooo crappy :-) ) trouble.
The FTP command that must be supported is REST.
From http://www.nsftools.com/tips/RawFTP.htm:
Syntax: REST position
Sets the point at which a file transfer should start; useful for resuming interrupted transfers. For nonstructured files, this is simply a decimal number. This command must immediately precede a data transfer command (RETR or STOR only); i.e. it must come after any PORT or PASV command.