I'm already familiar with Azure Artifacts, and lately I've been trying to optimize billing expenses. Since Azure Artifacts charges per/GB, I've been wanting to know, does the Universal Packages feature attempt to optimize storage usage by only storing the differences between one version of the package and the next version of the package?
does the Universal Packages feature attempt to optimize storage usage by only storing the differences between one version of the package and the next version of the package?
Agree with Jonathan. Customers are billed for the full size of each artifact stored on the service
That because Azure Artifacts could not intelligently extract deltas from your package version 2 (Sometimes we only modify the contents of the file.) unless we only package deltas when we pack the package. But in this case, this deltas package should be another package instead of package version 2 (It does not contain files from package version 1.).
On the other hand, when we use the package version 2, we still could use the package 1 independently. Azure Artifacts still provides services for package version 1, so it is reasonable to pay for it. If you do not want to billed for the full size of each artifact stored on the service, you could delete the package version 1 after upload the package version 2.
Note: Every organization can use up to 2 GB storage for free. Additional storage usage is charged according to tiered rates starting at $2 per GB and decreasing to $0.25 per GB:
Rate card
0 - 2 GB = Free
2 - 10 GB = $2 per GB
10 - 100 GB = $1 per GB
100 - 1,000 GB = $0.50 per GB
1,000+ GB = $0.25 per GB
So, if your package does not exceed the GB boundary, there will be no additional charges.
Hope this helps.
Customers are billed for the full size of each artifact stored on the service, regardless of how we physically store it.
Related
To give a bit background information on whats going on, few months back the company I'm employed with purchased Odoo ERP system and did a couple of custom modifications to it using a third party company. After the development developers pushed the code into a GCP VM instance. However my company was not using this release at all. 6 months after management wants to remove the existing GCP VM and transfer the source code of Odoo into Github to save the infrastructure cost.But I have absolutely no idea how to do this as I'm not coming from an infrastructure background. I can see a VM sitting in GCP with below specification.
Machine type: custom (8 vCPUs, 16 GB memory)
Reservation: Automatically choose
CPU platform: Intel Haswell
Zone: us-central1-a
Operating system: Ubuntu
It would be really helpful if someone can point me at the correct direction
GitHub is for hosting program sources only. You cant run your odoo from there. You may migrate to odoo.sh (And code to GitHub)
You need access to your modifications, source code, and database files. Then you can upload your source to GitHub. After that, buy an instance to odoo.sh and upload your database backup there.
And if you are interested in how to get files from GCE, then there should be information for it:
https://cloud.google.com/compute/docs/instances/transfer-files
We are using mta to structure our application consisting of multiple (11 nos) micro-services (Java + Nodejs + Python).
Using the SAP Cloud SDK Pipeline, we are packaging the aforesaid micro-services into mtar file and then uploading the generated mtar to Transport Management landscape which subsequently deploys the application to SAP CLoud Foundry.
The issue here is with the large size of the mtar generated. The mtar file size in our case ranges between 900 MB to 1.2 GB and that is a pain point as it exceeds the 400 MB limit set in TMS upload.
This fat size of the mtar is mainly because of the packaging of the node_modules (around 350 MB each) in the mtar for the nodejs (2 nos) microservices.
We understand that the mtar size limit to TMS upload can be configured but looking for suggestions on the best practices to optimize the size of the mtar file.
Appreciate if someone can guide us here on the efficient ways to handle the node_modules in the mtar and thereby optimize the large size of the mtar file.
I'm trying to develop an Azure DevOps extension similar to the official AWS Toolkit for Azure DevOps Extension.
What I'm struggling with is the fact that the aws-sdk is 45MB in size, but Azure DevOps extensions are limited to a 26MB upload size.
How does the official extension get their references to the aws-sdk working? It doesn't exist anywhere in the vsix extension package when I unzip it.
Has anyone else ran into this file size restriction problem before?
We limit VSTS extension sizes to 25 Mb (we can go up to max 100 MB)
and allow exceptions only in certain cases. We worked with the
publisher here to understand their requirement and increased the size
limit as an exception for their publisher.
Take a look at this similar question here: The extension package size exceeds the maximum package size '26214400 bytes'
Before increasing package size, Could you please optimize. Suggestions:
Can the common dependences, if any, be stated once in the extension
package?
Fetch things at runtime or during install time rather that providing
it within the package. Consider using the tool installer lib to pull
tool dependencies at runtime. Using the lib offers benefits where the
tool is cached by version so for private agents, it won't get
downloaded every build. We made it a lib so it can be used outside of
tool installer tasks. The one consideration, is the task will not
work in disconnected scenarios (no internet). That should just be
in the description / docs for the task.
Some customers have had good success with WebPack to tree shake their
dependencies in their tasks.
If you still blocked, you have to contact our VS Marketplace Support Team and try to extend the limitation.
I'm working on a Service Fabric application that is deployed to Azure. It currently consists of only 5 stateless services. The zipped archive weighs in at ~200MB, which is already becoming problematic.
By inspecting the contents of the archive, I can see the primary problem is that many files are required by all services. An exact duplicate of those files is therefore present in each service's folder. However, the zip compression format does not do anything clever with respect to duplicate files within the archive.
As an experiment, I wrote a little script to find all duplicate files in the deployment and delete all but one of each files. Then I tried zipping the results and it comes in at a much more practical 38MB.
I also noticed that system libraries are bundled, including:
System.Private.CoreLib.dll (12MB)
System.Private.Xml.dll (8MB)
coreclr.dll (5MB)
These are all big files, so I'd be interested to know if there was a way for me to only bundle them once. I've tried removing them altogether but then Service Fabric fails to start the application.
Can anyone offer any advice as to how I can drastically reduce my deployment package size?
NOTE: I've already read the docs on compressing packages, but I am very confused as to why their compression method would help. Indeed, I tried it and it didn't. All they do is zip each subfolder inside the primary zip, but there is no de-duplication of files involved.
There is a way to reduce the size of the package but I would say it isn't a good way or the way things should be done but still I think it can be of use in some cases.
Please note: This approach requires target machines to have all prerequisites installed (including .NET Core Runtime etc.)
When building .NET Core app there are two deployment models: self-contained and framework-dependent.
In the self-contained mode all required framework binaries are published with the application binaries while in the framework-dependent only application binaries are published.
By default if the project has runtime specified: <RuntimeIdentifier>win7-x64</RuntimeIdentifier> in .csproj then publish operation is self-contained - that is why all of your services do copy all the things.
In order to turn this off you can simply add SelfContained=false property to every service project you have.
Here is an example of new .NET Core stateless service project:
<PropertyGroup>
<TargetFramework>netcoreapp2.2</TargetFramework>
<AspNetCoreHostingModel>InProcess</AspNetCoreHostingModel>
<IsServiceFabricServiceProject>True</IsServiceFabricServiceProject>
<ServerGarbageCollection>True</ServerGarbageCollection>
<RuntimeIdentifier>win7-x64</RuntimeIdentifier>
<TargetLatestRuntimePatch>False</TargetLatestRuntimePatch>
<SelfContained>false</SelfContained>
</PropertyGroup>
I did a small test and created new Service Fabric application with five services. The uncompressed package size in Debug was around ~500 MB. After I have modified all the projects the package size dropped to ~30MB.
The application deployed worked well on the Local Cluster so it demonstrates that this concept is a working way to reduce package size.
In the end I will highlight the warning one more time:
Please note: This approach requires target machines to have all prerequisites installed (including .NET Core Runtime etc.)
You usually don't want to know which node runs which service and you want to deploy service versions independently of each other, so sharing binaries between otherwise independent services creates a very unnatural run-time dependency. I'd advise against that, except for platform binaries like AspNet and DotNet of course.
However, did you read about creating differential packages? https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-application-upgrade-advanced#upgrade-with-a-diff-package that would reduce the size of upgrade packages after the initial 200MB hit.
Here's another option:
https://devblogs.microsoft.com/dotnet/app-trimming-in-net-5/
<SelfContained>True</SelfContained>
<PublishTrimmed>True</PublishTrimmed>
From a quick test just now, trimming one app reduced the package size from ~110m MB to ~70MB (compared to ~25MB for selfcontained=false).
The trimming process took several minutes for a single application though, and the project I work on has 10-20 apps per Service Fabric project. Also I suspect that this process isn't safe when you have a heavy reliance on dependency injection model in your code.
For debug builds we use SelfContained=False though because developers will have the required runtimes on their machines. Not for release deployments though.
As a final note, since the OP mentioned file upload being a particular bottleneck:
A large proportion of the deployment time is just zipping and uploading the package
I noticed recently that we were using the deprecated Publish Build Artifacts task when uploading artifacts during our build pipeline. It was taking 20 minutes to upload 2GB of files. I switched over the suggested Publish Pipeline Artifact task and it took our publish step down to 10-20 seconds. From what I can tell, it's using all kinds of tricks under the hood for this newer task to speed up uploads (and downloads) including file deduplication. I suspect that zipping up build artifacts yourself at that point would actually hurt your upload times.
Is there a known solution for splitting build process cross the network machines?
Use case:
We are an average software development company. We own around 50 development workstations (Quad Core 2.66Ghz, 4 GB ram, 200 GB raid). No need to tell that at any single moment not every machine is loaded to the max.
There are 5 to 15 projects running simultaneously at any single moment. Obviously all of them are continuously build on server, than deployed to proper environment. Single project build is taking from 3 to 15 minutes.
The problem: Whenever we build 5 projects in a row the last project is going to be ready after around 25 - 50 minutes. Building in parallel does not solve the problem (build is only a part of the game, than you need to deploy, run tests etc.)
YES the correct solution is to add another build server, but "That involves buying new Expensive hardware, and we already spent a lot!". Yea, right(damn them)!
Anyway. What about splitting build among developers workstation? Lets say whenever we need to build project "A" we check 5 workstations and start build on all that are not overloaded. The build can be canceled by a developer if he really needs all the power of his machine as long as there is at least 1 machine that is still building. After build is finished deployment can be performed to a proper environment (hosted on some server, not on workstation :) ). The bigger the company the more this makes sense to me.
Anyone tried something like this? Are there any good practices? Any helpful software?
(90% of the projects are .net C#, platform - Windows)
You can also check our Parabuild at http://www.viewtier.com - it allows to designate a set of machines as a build farm. Works practically on any platform. It looks like that's what you are looking for.