I have a Github repo that includes some large graphical assets. These assets result in failed deploys to Netlify due to Netlify's size restrictions. Is there any way I can keep these files within the Github repo but exclude them from Netlify deploys, in the same way I could use a .slugignore file when deploying to heroku.
Netlify doesn't really have explicit size restrictions, though uploads of files >20MB may fail. Are your files bigger than that? If so, hosting them on Netlify's CDN also doesn't make sense as the CDN edge cache will ignore them and they'll load slowly for browsers anyway.
To not deploy them, the most straightforward way is to remove them after your build, something like this:
npm run build && rm dist/hugefile1.jpg dist/subdir/hugefile2.pdf
You can get fancier and use a file to list them or just look for everything huge. Warning - something huge that it DOES make sense to host is your sourcemap (if you use one), so watch out for what this might catch!
npm run build && find dist -type f -size +20M
Effectively - you can do anything you could do in a shell script. NB: You need to make sure that your build pipeline fails if any necessary step fails - this is why the examples show && to chain commands rather than ; (build could fail, find succeed, and we publish an empty site!).
More details here: https://www.netlify.com/blog/2016/10/18/how-our-build-bots-build-sites/ and you can test out your scripts using the methodology described here: https://github.com/netlify/build-image#testing-locally
Related
I am working on a project with multiple people, a website application which requires webpack to be built, uglified, concatenated into a few files e.g. app.min.js, style.min.css etc. - As a result of this, in an effort to prevent merge conflicts we recently added the build folder to .gitignore, under the assumption that we would be able to build during deployment.
When pushing to the Master branch, we automatically "deploy" through Semaphore CI (similar to Travis) which runs composer install, npm install, and finally "npm run build" which triggers the webpack build. This is all built and then tested on the CI side of things, and then Semaphore automatically deploys to Amazon's Elastic Beanstalk where our application is hosted.
The problem with this is, it seems Semaphore doesn't upload the build it's just tested, but rather the Master branch itself which has no built JS or CSS. I'm wondering if there's a way to push these built files to deployment as well, or if running the entire build process AGAIN on Elastic Beanstalk is the only route. It seems unnecessary to have to do that process essentially 3 times, locally, CI, and then deployment. Every time a step like this is needed on EB the actual re-instantiation time gets longer, which I'd like to keep as short as possible.
Obviously if building it a 3rd time on EB is the only way to go about this then I'll have to, just wondering if there are better solutions for this whole workflow.
I haven't worked with Semaphore CI, but you might be able to use an .ebignore file.
If you create one, the cli will use that instead of your .gitignore file.
I find in some deployment situations you want the inverse of your .gitignore (all compiled, no src). It essentially lets you pick the files from your project directory that you want to deploy, in the same way as the .gitignore file.
Edit: I just noticed the documentation on aws is lacking. It only mentions file exclusion, but you can include files too.
Edit 2: I don't think Semaphore supports the use of .ebignore, so right now this solution isn't of any use. :(
I just had a great first experience with https://deploybot.com/. The can deploy directly to elastic beanstalk. It might be interesting or you.
I'm deploying my app to ElasticBeanstalk. I'm using an .ebignore file because there are files that I do not want to check into git, but I do want deployed with the app(like application secrets, config vars, etc). The issue I'm facing is that when using an .ebignore, the deploy takes FOREVER. I've used the --verbose flag, and I can see that it is recursing my entire node_modules directory and skipping each file individually. When I deploy by using .gitignore, it becomes very fast.
Has anyone else experienced this? How do I speed up this process?
So, I have a webapp I am creating using the 3 muskateers yeoman, grunt and bower.
My questions are:
What is best practice when it comes to uploading my webapp into a git/mercurial repo? Do I include the entire project? What about directories like 'node_modules' or 'test', etc?
Also, when deploying to live production site: Will my 'dist' folder be what I should be uploading?
With research yielding no results (I could be searching the wrong things?).. I'm a bit new to this process so any feedback is greatly appreciated. Thanks!
You should always commit all of your yeoman, grunt, and bower config files.
There are two schools of thought on committing the output they produce or dependencies they download:
One is, you should upload everything needed for another user to deploy the web app after cloning the repository, without performing any additional operations. The idea is, dependencies may not exist anymore, network connections might be down, etc.
Another is, keep the repository small and don't commit node_modules, etc, since they can be downloaded by the user.
As far as the dist folder goes, yes you'll be uploading it to your server, as it contains all of your minified files. Whether or not you want to commit it to the repository is a separate question. You might let the user build every time, assuming they can get all the dependencies one way or another (from above choice). Or you might want to commit it to tag it with a release version along with your source code.
There's some more discussion on this here: http://addyosmani.com/blog/checking-in-front-end-dependencies/
I'm new to Capistrano and struggling a little to get started. A brief description of what I need to do:
git pull the latest code from our git repo, on a central build server. This build server's environment matches the deployment environment exactly. I need the code to be built here. I don't want to deploy a binary that was built on a Mac laptop, for example.
compile the binary on this machine.
deploy it from this machine to all the target machines.
There is a shared user we can all SSH into on the build machine to do the builds.
The build machine is behind a gateway machine, not directly accessible.
All of the deployment target machines also have this shared user and are also behind the gateway.
The deployed binary is a single executable, and there is an init script on the target machines. After deploying the binary and changing the symlink to it, restart the service via the init script.
Everyone has appropriate SSH keys and agent forwarding for all necessary tasks.
So in principle it seems rather simple, but Capistrano seems opinionated and a bit magical. As a result I'm not sure how to accomplish all of this. It seems like it wants to check out my code and copy it to the remote machines, for example without building it first.
I think I need to ignore all of Capistrano's default smarts and just make it run some shell commands on the appropriate servers. In pseudo-code:
ssh buildmachine via gateway "cd repo && git pull && make"
ssh targetmachine(s) via gateway "scp buildmachine:repo/binary .; <mv && symlink>; service foo restart"
Am I even using the right tool for the job? It seems a lot like a round peg in a square hole.
Can someone explain to me what the contents of the Capistrano configuration files should be, and what cap commands I'd run to accomplish this?
BTW, I've searched around and looked at questions like deploying with capistrano with remote git repo but without git running on production server and From manual pull on server to Capistrano
The question is rather old, but you never know when someone steps onto it in need of information...
First and formost, consider that Capistrano might just not be the right tool for the job you want to do.
That said, it is not impossible to accomplish what you expect. While in projects that deploy large amount of files and modify them (like css/js minify, js builds etc.) I would avoid it, in your case, you can consider runing a "deployment repository" and configure it in capistrano as the source. Your process would look like this :
run the local build with whatever tools you need
upload resulting binary to a deployment repository
run capistrano that will connect to application servers, fetch fresh binary from repository, perform any server side tasks required and symlink to "current"
As a side effect you end up with full history of deployed binaries
We are a small team of 4 developers working on a web application. We use trac+svn on a shared server for version control and ticketing and we are happy and satisfied with this. The same shared server also hosts our web application. The application itself is a Perl CGI application that uses CGI::Application and a moderate number of standard (CPAN) and custom Perl modules that are installed in the usual (/usr/lib/perl...) and a few unusual locations (/home/user/lib/perl..). While the broad details might be irrelevant, the most important point is that the location/layout of libraries on our development machines is different from that on the production (shared) server. We have to live with this as a given. The library layout is identical on all development machines though.
Here is a typical, but clearly sub-optimal work-cycle that my colleagues and I follow:
Code and test on development machines
Checkout/Commit/Update our code onto the SVN
Periodically "svn export" onto the appropriate DocumentRoot of the server
Hand edit the exported tree to set the library includes match the library layout on the server
Test application on live server, raise tickets for each other
Go to 1
Clearly there must be a better way and would appreciate hearing from others who might be handling this better than we are. For example is there a way to svn export and fix the library locations in an automated way? Or is there some completely different way to handle this situation than we have been doing so far.
Thank you for your attention
You should have scripts that do this for you that can be run from a local box. Mine always look something like:
$> checkout from source or copy from working
$> run sed/perl -pi/copy to convert configs to the production values
(ie cp production.config myconfig)
$> upload to web server (rsync/ssh/ftp/etc)
$> ssh $SERVER migrate_db, set permissions, run unit tests, etc
The last one requires ssh access which I always look for but everything else can be done locally. You'd usually have a set of dev configs and a set of production configs (or a script to convert from dev to production
One step uploads are always a really good idea.
Keep a config file (e.g. config.pl) that stores all the system-dependant paths and variables. Then set the svn:ignore property on this file so that it is never commited. This will allow you to easily keep a local configuration script per system that is separate from the commited tree.
If you don't have the possibility of mirroring your development server in production, why can't you mirror your production server in development? That might take some reconfiguring, but what's the risk? Everything's checked into svn.
But maybe that really, really isn't an option for you. My preference for deploying web applications is to do an svn checkout and then run a symlinking script. The idea is that you write a system of rules that logically maps the contents of one folder to the contents of another. Of course, if you drop folder symlinks in your document root, you have to tell Apache to follow them.
Frankly, the absolute safest scenario would be to set up a virtual machine that you can configure exactly like your production machine. This way you can actually test the contents of your deployment script and submit tickets. Then, when the problem is discovered, you modify the script to make it more likely that development deployment will follow the new and improved procedure.
And, as a side note: I much prefer to use svn checkouts in place rather than svn exports. It shouldn't be hard (especially if you use a deployment script) to make sure that apache or whatever your web server is doesn't have permission on the .svn folders. Ideally, anything you can do to make an svn rollback a one-line command is absolutely key.
If this is a Linux box you could write a cron job to take care of this for you. You could use sed/awk to replace the needed strings in the code and svn export works fine from a cron job. You would need to maintain the script but it seems faster than doing it by hand every time.
For the hand-editing part, I would have a separate branch in Subversion for the local modifications you need. The developers commit into trunk and when you need to deploy, use 'svn merge' or svnmerge.py to merge changes from trunk to the branch.
After creating the branch the first time, make your local modifications in there.
On the servers, have the directories in DocumentRoot and /usr/lib/perl and /home/user/lib/perl be Subversion working copies checked out from the branch.
Do not use svn export, just have a checkout so you can 'cd /usr/lib/perl; svn up'.
The one thing to be careful about is not exposing your .svn directories in DocumentRoot, use this:
# Prevent any access to .svn directories.
<DirectoryMatch "^/.*/\.svn/">
Options None
AllowOverride None
Order allow,deny
Deny from all
</DirectoryMatch>
Having working copies deployed in DocumentRoot is also nice, if you need to rollback a change, just 'svn up -r PREV'.
Codesion offers enterprise grade subversion and trac on demand. In addition, we now have the ability to one-click publish / deploy your code via ftp, scp, rsync and many other methods. This will be an easy and quick way for you to accomplish what you are trying to do.
See our Codesion Publisher features:
https://help.codesion.com/View.jsp?procId=01fabe5e83381dda4edda959b97b2c5b