I am often working with websites to which I have no code access but for which I need to test javascripts, ajax calls and other resources in locally modified versions.
As an example I needed to test a new creative code originally coming from a DoubleClick server. Instead of working directly on DoubleClick and loose a lot of time waiting for my changes to take effect I needed to manipulate a local copy of that javascript resource.
So I changed my /etc/hosts to point the DoubleClick server to localhost where I run a local web server. This way I was able to test a local script instead of the original.
Of course this redirected all resources and not only the one i was interested in what often results in either weird behaviour as resources become unavailable or a lot of effort on my part to make them available.
Is there a way to replace a specific URL by redirecting it to a local file/URL and let all the others from the same origin be untampered?
Related
I manage a website which is built from a GitHub repository via an action which commits a live version to a certain branch, and then the webserver routinely checks if there are any updates on this branch and, if so, pulls them down to its public_html directory. This then serves the website on some domain, say
example.com.
For various (practically immutable) reasons, there are individual webpages that are "built" from other individual repositories — here I say "built" because these repositories are almost always just some .html files and such, with little post-processing, and could thus be served directly via GitHub pages. I want these to be served at example.com/individual-page. To achieve this, I currently have a GitHub action which transfers the files via FTP to a directory on the webserver which is symlinked inside public_html, thus making all the files accessible.
However, it now occurs to me that I could "simply" (assuming this is even possible — I imagine it would need some DNS tweaking) activate GitHub pages on these individual repositories, set with the custom domain example.com, and avoid having to pass via FTP. On one hand, it seems maybe conceptually simpler to have public_html on the webserver only contain the files coming from the main website build, and it would be simpler to make new standalone pages from GitHub repositories; on the other hand, it seems like maybe "everything that appears on example.com should be found in the same directory" is a good idea.
What (if any) is the recommended best practice here: to have these individual pages managed by GitHub pages with custom domains (since they are basically just web previews of the contents of the repositories), or to continue to transfer everything over to the webserver in one directory?
In other words maybe, is it a "good idea" to partially host your website with GitHub pages? Is this even possible with the right DNS settings?
(I must admit, I don't really understand what exactly my browser does when I navigate to example.com/individual-page, and what would happen if such a directory existed in my webserver and also GitHub pages was trying to serve up a webpage at this same address, so I guess bonus points if you want to explain the basics!)
The DNS option you describe doesn't work for pages.
While you can use a CNAME record to point your domain to another domain or an A record to point your domain to an IP address, DNS doesn't handle individual pages (as in example.com/a). It would work if each page was, for instance, a.example.com, but that's not a page, it's a whole different website.
What you can do, however, is include those other repositories as submodules of your repository, and then everything works without any DNS magic (except the simple CNAME record, which isn't magic).
It would be a good idea to implement this described solution, as it's the simplest. In any case, as long as your current solution works automatically without issues and the hosting cost isn't an issue, I don't see any need to take the time to implement a new solution.
If you want to only serve some files or pages from the submodules, you can have build actions and serve a specific directory.
I must admit, I don't really understand what exactly my browser does when I navigate to example.com/individual-page
Your browser requests the DNS records of your domain (example.com), in this case the A record (since this is the root domain). This A record gives an IP address, which your browser uses to request the page. As you can see, pages aren't handled by DNS at all.
That means only one machine handles all requests for a domain (which doesn't mean it can't delegate some pages to another machine, but that's more complex). That means it's impossible to have "directory conflicts", because either your server handles the request or GitHub does. Your browser doesn't check if the page exists at server A and if not, if it exists at server B.
I have developed a web app that uses Nextjs app for the frontend and a Python (Django) API server as the backend. Most of my front-end pages have API calls to my backend server (in ComponentDidMount or in response to user actions like button clicks).
I want to deploy this app to my server. I am using Nginx as the reverse proxy. The backend deployment is sorted. I am confused about deploying the nextjs app.
After reading the docs I figured there are 2 ways to do this:
Run next build and then next start. This will start a nodejs server on port 3000. I can channel traffic to this port using Nginx.
Run next export. This will generate an out directory. I can channel incoming traffic to this directory using a reverse proxy like Nginx.
Which of the 2 options should I use? What are the differences?
Answering my own question with the help of the answer I received on the NextJS discussions forum link here
Basics of next build, start and export
next build builds the production application in the .next folder. You need to run this command irrespective of whether you want to run next start or next export.
After building, next start starts a Node.js server that supports hybrid pages, serving both statically generated and server-side rendered pages.
next export will export all your pages to static HTML files that you can serve with any host. This is similar to what create-react-app does, but you can still build dynamic pages at build-time with exportPathMap.
Note: Statically generated pages, using next export, are still reactive i.e. any dynamic JS code, which updates your pages at run time, will continue to run as usual (like any other ReactJS app). Next.js will hydrate your application client-side to give it full interactivity. Its just that the dynamic section of the page will only be rendered in the browser at run time and hence it won't be available to search engine crawlers.
When to use next export?
next export is recommended if you have some dynamic pages which need to some data to be fetched only at 'build' time. It is perfect for pages like landing pages, blogs, news articles etc, or for other kinds of pages which are updated only during code builds. You can setup a background task to build your code at regular intervals if your page data needs to be updated more frequently. This will speed up your 'first paint' speed and also provide the entire page data to search engine crawlers for indexing.
When to use next start?
If you do not need any data fetching at build time i.e. your pages are rendered dynamically at run time then you should use next build and then next start. This will also generate static HTML for sections of your page which are not dynamic. This will speed up the 'first paint' time of your pages. See this Automatic Static Optimization.
next export, also produces an optimized production ready build, but it builds fully static app, which means only html, css and javascript but no server-side code.
Since there is no server side code, you do not need a server to host your app. This makes hosting your app easier and cheaper because you dont need to maintain and scale a server. There are plenty of hosts in the market which offers very good pricing.
Since next export builds a static app, it means you cannot use api routes, server-side functions therefore you cannot revalidate. Revalidation means, your app will be checking database for a certain of time that you specify. Let's say you have a blogging app, you save your blog posts to database, with server side coe, you get those on server side, pass them to the client and client renders the posts and with revalidation option set, your app would automatically check the database if the content changed. Since you cannot use this feature, anytime content of your app changes, you have to redeploy your app.
So I'm using a very simple CDN service. You point to your website and if you call it through their HostName they'll cache it for you after the first call.
I use this for all my static content, like JavaScript files and images.
This all works perfect - and I like that it has very little maintenance or setup cost.
Problem starts when rolling out new versions of JavaScript files. New JavaScript files automatically get a new hash if the files changes.
Because roll out over multiple instances is not simultaneously a problem occurs though. I tried to model it in this diagram:
In words:
Request hits server with new version
Requests Js file with new version hash
CDN detects correctly that the file is not cached
CDN requests the original file with the new hash from the load balancer
loadbalancer serves request of CDN to a random server - accidently serving from a server with the old version
CDN caches old version with the new hash
everyone gets served old versions from the CDN
There are some ways I know how to fix this - i.e. manually uploading files to a seperate storage with the hash baked in, etc.
But this needs extra code and has more "moving parts" that makes maintenance more complicated.
I would prefer to have something that works as seamlessly as the normal CDN behavior.
I guess this is a common problem for sites that are running on multiple instances, but I can't find a lot of information about this.
What is the common way to solve this?
Edit
I think another solution would be to somehow force the CDN to go to the same instance for the .js file as the original html file - but how?
Here are a few ideas from my solutions in the past, though the CDN you are using will rule out some of these:
Exclude .js files from the CDN Caching Service, prevents it being cached in the first place.
Poke the CDN with a request to invalidate the cache for a specific file at the time of release.
In your build/deploy script, change the name of the .js file and reference the new file in your HTML.
Use query parameters after the .js file name, which are ignored but cached under a different address reference, e.g. /mysite/myscript.js?build1234
The problem with this kind of issues is that the cache control resides on the browser side, so you cannot do too much form the server side.
The most common way I know is basically the one you mention about adding some hash to the file names or the URLs you use to get them.
The thing is that you should not do this manually. You should use some web application builder, like Webpack, to automate this process and it will depend on the technologies you are using. I saw this for the first time using GWT 13 years ago, and all the last projects I worked with, using AngularJS or React, had been integrated with builders that does what you need automatically.
Once it's implemented, your users will get the last version, and resources will be cached correctly to speed up your site.
If you can also automate the full pipeline to remove the old resources from the CDN once the expiration configured on them have been reached, you touched the sky.
I fixed this in the end by only referencing to the CDN version after a few minutes of runtime.
So if the runtime is less then 5 minutes it refers to:
/scripts/example.js?v=351
After 5 minutes it refers to the CDN version:
https://cdn.example.com/scripts/example.js?v=351
After 5 minutes we are pretty sure that all instances are running the new version, so that we don't accidently cache an old version with the new hash.
The downside is that on very busy moments you don't have the advantage of the CDN if you would redeploy, but I haven't seen a better alternative yet.
For example, I have a website with various types of information. If that goes down I have a copy of the same website the users use on a local webserver, like Apache or IIS on the client. They use this local version until the Internet version returns. They can have no downtime, in other words.
The problem is that over time the Internet version will change while the client versions will remain the same unless I touch each client's machine to make the updates. I don't want to do that.
Is there a good way to keep my client up to date so that when I make a change on the server the client gets a copy so they can run it locally if needs be?
Thank you.
EDIT: do you think maybe using SVN and timely running of the update by the clients would work?
EDIT: they'll never ever submit anything. It's just so I don't have to update the client by hand, manually going to the machine. they're webpages that run in case the main server is down.
I will go for Git over SVN because of its distributed nature. Gives you multiple copies of code; use it along with this comment's solution:
Making git auto-commit
to autocommit.
Why not use something like HTTrack to make local copies of your actual internet site on each machine, rather then trying to do a separate deployment. That way you'll automatically stay in sync.
This has the advantage that if, at some point, part of your website is updated dynamically from a database, the user will still be able to have a static copy of the resulting site that is up-to-date.
There are tools like rsync which you can use periodically to sync the changes.
We have an internal web system that handles the majority of our companies business. Hundreds of users use it throughout the day, it's very high priority and must always be running. We're looking at moving to ASP.NET MVC 2; at the moment we use web forms. The beauty of using web forms is we can instantaneously release a single web page as opposed to deploying the entire application.
I'm interested to know how others are deploying their applications whilst still making them accessible to the user. Using the deployment tool in Visual Studio would supposedly cause a halt. I'm looking for a method that's super quick.
If you had high priority bug fixes for example, would it be wise to perhaps mix web forms with MVC and instead replace the view with a code-behind web form until you make the next proper release which isn't a web form?
I've also seen other solutions on the same server of having the same web application run side-by-side and either change the root directory in IIS or change the web.config to point to a different folder, but the problem with this is that you have to do an entire build and deploy even if it were for a simple bug fix.
EDIT: To elaborate, how do you deploy the application without causing any disruption to users.
How is everyone else doing it?
I guess you can run the MVC application uncompiled also? and just replace .cs/views and such on the run.
A websetup uninstall/install is very quick, but it kills the application pool.. which might cause problem. Depending on how your site is built.
The smoothest way is to run it on two servers and store the sessions in sql server or shared state. Then you can just bring S1 down and patch it => bring s1 back up again and bring S2 down => patch S2 and then bring it up again. Al thought this might not work if you make any major changes to the session parts of the code.
Have multiple instances of your website running on multiple servers. The best way to do it is to have a production environment, a test environment, and a developement environment. You can create test cases and run the load every time you have a new build, if can get through all the tests, move the version into production ;).
You could have two physical servers each running IIS and hosting a copy of the site. OR you could run two copies of the site under different IIS endpoints on the SAME server.
Either way you cut it you are going to need at least two copies of the site in production.
I call this an A<->B switch method.
Firstly, have each production site on a different IP address. In your company's DNS, add an entry set to one of the IPs and give it a really short TTL. Then you can update site B and also pre-test/warm-up the site by hitting the IP address. When it's ready to go, get your DNS switched to the new site B. Once your TTL has expired you can take down site A and update it.
Using a shared session state will help to minimise the transition of users between sites.