I finally managed to automate our release process using Desired State Configuration with the Azure PowerShell SDK methods, in particular the Publish-AzureVMDscConfiguration -> Set-AzureVMDscExtension -> Update-AzureVM combo.
After thinking for a while in a way to send my build outputs somewhere accessible by the VM, I ended up with the strategy of appending my build drops in the configuration package that gets uploaded to Azure Storage.
My problem now is that as soon as the PowerShell DSC Extension in the VM starts downloading that package, it's memory consumption goes through the roof. When I open task manager, I can see the newly created PowerShell process going from 30 or so megabytes, to 300, and then to 1.3GB, completely ruining my VM.
Yesterday afternoon, I left work and let it processing, but when I logged into the VM today, the inner zip file, containing my build outputs, had 0 bytes in the DSCWork folder. My problem is that even if it worked in the end, it is taking a very long time and making my VM useless... I can't even change between windows in remote access, since the machine is completely stuck at 100% RAM usage.
Why is PowerShell taking so much memory and time to download my configuration package? It only has 60MB zipped, and roughly 200MB unzipped. Is there something I can do to prevent that from happening?
UPDATE:
I tested it just now and it finally finished correctly. Took more than a full hour, but the files are there... This is unacceptable though.
This issue should be resolved in the next iteration of the extension. In the meanwhile you may want to consider uploading your build content to a blob separate from your configuration ZIP package (you can use Set-AzureStorageBlobContent for this).
Then you can use either the remote file or script resources in your original configuration to download the blob. Be sure to add the appropriate dependencies in your configuration so that the blob gets downloaded before you use it.
configuration DownloadSample
{
Import-DscResource -Module xPSDesiredStateConfiguration
xRemoteFile Download
{
Uri = 'https://....blob.core.windows.net/windows-powershell-dsc/foo.zip?sv=...'
DestinationPath = 'd:\tmp\download.zip'
}
}
Related
I am trying tWaitForFile component in Talend to watch for new created files. It seems to be working for local directory (I am using Windows 7).
However, when I point it to a shared folder like //ps1.remotemachine.com/Continents/Africa it doesn't work. It doesn't give me file creation signals like it gives for local directory.
Am I missing something?
Update:
In my testing so far, below are the observations for monitoring files on network path:
Talend tWaitForFile - Inconsistent results. Only gives notification sometimes. Majority of time, doesn't.
Java Nio WatchService - Tried this out of Talend solution. It does give notification for created files on network path. However, when the number of folders to be monitored on network path are too many, it starts missing events of some of the folders. In my case, it was around 100 folders to be monitored.
Hence, aborted both of above approaches and sticking on scheduler based running of Talend jobs.
Use
"\\\\ps1.remotemachine.com/Continents/Africa"
If you use the value from a context then you don't need to double "\"
And in the tWaitForFile
I need to download file from http using powershell, however, it never downloads the complete file. Also, please suggest, how can I download the whole folder from http site with all its content, I could able to see some examples on the site, however, since I am new to powershell, I am unable to completely make it work for me.
I need to download zip file(500 mb) from http to one of my server to install software after download.
$webclient=New-object system.net.webclient
$weclient.downloadfile("http://mysite/file.zip","c:\testfolder\file.zip")
zip file should download, however, it starts downloading, but never download complete 500 mb file, sometimes it downloads 10 mb, sometimes 20 mb file. I am using Windows 2012 standard for this purpose. If I try to download small files, it downloads completely, however not the 500 mb zip file.
New Answer:
Since this is a network issue, you could try a forever loop until the file you download is complete:
do{sleep 3;(New-Object Net.WebClient).DownloadFile('http://mysite/file.zip','C:\testfolder\file.zip')}while(!$?);
It essentially sleeps for 3 seconds in the beginning, attempts to download the file using the webclient. The while condition is to check if the webclient command returns a false which !$? is short hand for and then repeats itself if it is false. Now, if the webclient command is completing successfully and you're still getting an incomplete file, there might be something else in play. But we'll figure it out once we get there :).
Let me know if this works, this should retry until it is finished.
Old Answer:
If the website you're downloading from is using TLS, you should enable it in your current session using the following:
``` [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12```
Though it seems the website might be only ```http```, this first part might not be needed.
Next, I recommend using ```Invoke-WebRequest``` to download the file. It's simpler than using the webclient.
```Invoke-WebRequest -Uri "http://mysite/file.zip" -Outfile "c:\testfolder\file.zip"```
Try this out and let us know if it works.
I have to think this is a solved issue but I am just not getting it to work. So I have come to you StackOverflow with this issue:
I have a windows server 2016 machine running in amazon ec2. I have a machine.ps1 script in a config directory.
I create an image of the box. (I have tried with checking noreboot and unchecking it)
When I create a new instance of the image I want it to run machine.ps1 at launch to set the computer name and then set routes and some config settings for the box. The goal is to do this without logging into the box.
I have read and tried:
Running Powershell scripts at Start up
and used this to ensure user data was getting passed in:
EC2 Powershell Launch Tools
I have tried setting up a scheduled task that runs the machine.ps1 on start up (It just hangs)
I see the initializeInstance.ps1 on start up task and have tried to even coop that replacing the line to run userdata with the line to run my script. Nothing.
If I log into the box and run machine.ps1, it will restart the computer and set the computer name and then I need to run it once more to set routes. This works manually. I just need to find a way to do it automagically.
I want to launch these instances from powershell not with launch configurations and auto scale.
You can use User data
Whenever you deploy a new server, workstation or virtual machine there is nearly always a requirement to make final changes to the system before it’s ready for use. Typically this is normally done with a post-deployment script that might be triggered manually on start-up or it might be a final step in a Configuration Manager task sequence or if you using Azure you may use the Custom Script Extension. So how do you achieve similar functionality using EC2 instances in Amazon Web Services (AWS)? If you’ve created your own Amazon Machine Image (AMI) you can set the script to run from the Runonce registry key, but then can be a cumbersome approach particularly if you want to make changes to the script and it’s been embedded into the image. AWS offers a much more dynamic method of injecting a script to run upon start-up through a feature called user data.
Please refer following link for ther same:
Poershell User data
Windows typically won't let a powershell script call another powershell script unless it is being run as Administrator. It is a weird 'safety' feature. But it is perfectly okay to load the ps1 files and use any functions inside them.
The UserData script is typically run as "system". You would THINK that would pass muster. But it fails...
The SOLUTION: Make ALL of your scripts into powershell functions instead.
In your machine.ps1 - wrap the contents with function syntax
function MyDescriptiveName { <original script contents> }
Then in UserData - use the functions like this
# To use a relative path
Set-Location -Path <my location>
# Load script file into process memory
. <full-or-relpath>/machine.ps1
# Call function
MyDescriptiveName <params-if-applicable>
If the function needs to call other functions (aka scripts), you'll need to make those scripts into functions and load the script file into process memory in UserData also.
I have two drives A and B. Using a python script I am creating some files in "A" drive and I am running a powerscript which copies all the files in the drive A to drive B in the interval of 1 sec.
I am getting this error in my powershell.
2015/03/10 23:55:35 ERROR 32 (0x00000020) Time-Stamping Destination
File \x.x.x.x\share1\source\ Dummy_100.txt The process cannot access
the file because it is being used by another process. Waiting 30
seconds...
How will I overcome this error?
This happened is because the file is locked by running process. To fix this, download Process Explorer. Then use Find>Find Handle or DLL, find out which process locked this file. Use 'taskkill' to kill that process in commandline. You will be fine.
if you want to skip this files you can use /r:n that n is times of tries
for example /w:3 /r:5 will try 5 time every 3 seconds
How will I overcome this error?
If backup is, what you got in mind, and you encounter in-use files frequently, you look into Volume Shadow Copies (VSS), which allow to copy files despite them being ‘in use’. It's not a product, but a windows technology used by various backup tool.
Sadly, it's not built into robocopy, but can be used in conjunction with it. See
➝ https://superuser.com/a/602833/75914
and especially:
➝ https://github.com/candera/shadowspawn
It could be many reasons.
In my case, I was running a CMD script to copy from one server to another, a heap of SQL Server backups and transaction logs. I too had the same problem because it was trying to write into a log file that was supposedly opened by another process. It was not.
I ran many IP checks and Process ID checkers that I ran out of knowing what was hogging the log file. Event viewer said nothing.
I found out it was not even the log file that was being locked. I was able to delete it by logging into the server as a normal user with no admin privileges!
It was the backup files themselves by the SQL Server Agent. Like #Oseack said, there may have been the need to use another tool whilst the backup files themselves were still being used or locked by the SQL Server Agent.
The way I got around it was to force ROBOCOPY to wait.
/W:5
did it.
I am using a virtual machine to automate the execution of integration tests for a server-based product.
I am using "Windows XP Mode and Virtual PC" on a developer machine.
I am doing everything using PowerShell. I wish to:
mount the VHD (diskpart)
copy the release package onto the VHD file system
dismount the VHD (diskpart, again)
Start the VM
It is all fine as long as I don't do 2. If I change the VHD file system at all then 4. fails silently.
If I then go to the Virtual Machines on the Host and start the VM up using the GUI I get a warning:
"Inconsistency in virtual hard disk time stamp detected"
"The virtual hard disk's parent appears to have been modified"
I suspect there is a security feature in here (would make sense). But in my case this feature is not desirable.
Anyone know how to disable the timestamp checking or set the timestamp after I unmount the VHD (before?) ...?
EDIT: Look at the Startup2() options ... method takes one parameter, one of which says:
vmStartupOption_FixParentTimestampMismatch = 1
... from:
Microsoft method details
As per my edit - there is a Startup2() method that takes a parameter that says to ignore the timestamp.