I am working through my first Azure HDInsight tutorial. Can I do this without installing Azure Remote PowerShell on my local computer?
Can I use SSMS (2008R2) to run the PowerShell? My first attempt at that led me down the path of using a Database in Azure, but I do not think that is what I want to do (the tutorial describes setting Storage (not a Database) and then an HDInsight instance to interact with that Storage).
I am doing this tutorial: http://www.windowsazure.com/en-us/manage/services/hdinsight/get-started-hdinsight/
Thank you.
While you can use SQL Server and HDInsight together as part of a full pipeline, for the purposes of the getting started tutorial you want to think of them as two very different things.
The Storage referred to, is a standard Windows Azure Storage account, based on blobs. These then form a backing file system for the HDInsight cluster.
As far as using PowerShell goes, it is definitely the best, and easiest way to submit jobs to an HDInsight cluster. I would also recommend using a regular PowerShell console, or the PowerShell ISE to work with HDInsight as well, rather than the one available through SSMS, since the SSMS version won't load all the Azure modules by default.
There are other ways to submit jobs if PowerShell is not your thing (if you are on OS X or Linux for instance). You can use the REST API provided by WebHCAT (documentation). If you're on Windows, and prefer C# to PowerShell, you can also use the Windows Azure HDInsight Management Client from the Microsoft Hadoop SDK to submit jobs (available on codeplex and nuget). These will need you to break out Visual Studio and write a short console program to submit your job, so may be a bit heavy unless you're doing full on C# streaming Map Reduce, and so are already there.
If you're after a GUI based approach to job submission to HDInsight, you're out of luck at the moment, but your might like to check out what my team is working on at Red Gate, which will help you with submitting Hive and Pig jobs.
Related
I need to develop powershell script for automating vm patches updating in azure. I know we can do it by using azure update management but is there any other way to do it? I'm new to it please help.
You could refer to this Virtual Machines - Update to update a virtual machine.
PATCH https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/virtualMachines/{vmName}?api-version=2020-06-01
In addition, we could also do this in the UI and capture the request URL and request Body in the Network via F12, then convert it to power shell script.
As per this Microsoft Q&A platform question, you may use PSWindowsUpdate powershell module cmdlets.
And, if you are looking for a way to implement a way by removing dependency on Azure Automation and Azure Log Analytics then check Automatic VM guest patching (Preview).
Also, I would love to know the reason for trying to manually develop script for automating VM patches when the same is already available to us in the form of Update management solution (that is integrated with Azure Automation and Azure Log Analytics).
I have some .exe files which i need to install on VMs in azure using azure automation powershell DSC. Is this possible and how can it be done? The exe files are currently stored in an azure file share.
A similar question was asked here The answer provided was:
If you want to do the install the EXE remotely then the VM needs to be configured to allow for remote management - you can do this via WinRM. See: https://www.penflip.com/powershellorg/secrets-of-powershell-remoting/blob/master/accessing-remote-computers.txt and http://tarkus.me/post/64761019099/windows-azure-vms-remote-management
Setting this up is non-trivial (you need to ensure it's secure). You could also try adding an VM extension to the VM that will do the copy. Since the VM is already provision I think the only option here would be to use DSC though I've never added a DSC extension to a VM post-provisioning, it should work. That would require authoring and staging a DSC script in addition to adding the extension but definitely less complex than enabling remote management.
You should also consider Azure automation, depending on how large scale is your automation projects are. You can find more info and tutorials here
What are the differences between Azure PowerShell Commandlets and Command-Line Tools?
is there a difference in capabilities?
are new versions published at around the same time?
is one of them the 'preferred' way to manage Azure resources?
I am interested in managing web sites, VMs and cloud services from developer's machine and also from Ci server (TeamCity/MSBuild - starting / stopping VMs).
I've noticed that Command-Line Tools is just a small ~5MB download while PowerShell commandlets have several dependencies (~22MB) and June 2013 version still relies on Azure SDK 1.8 October 2012.
The PowerShell cmdlets work only in PowerShell (on Windows machines). They're great for automating several concurrent tasks (i.e. scripting out an entire environment), or carrying out single tasks.
The command-line tools are cross platform tools (work on Windows, Linux, etc.) These are written in node.js. They're not as good for scripting multiple actions, but are great for developers that prefer the command line to execute tasks (instead of going to the Azure management portal).
As far as I can tell, they are not on a synced release schedule. You can follow the progress of each via their github.com repositories (https://github.com/WindowsAzure/azure-sdk-tools-xplat and https://github.com/WindowsAzure/azure-sdk-tools).
It appears that Powershell is a wrapper for the node tools. If you get an error in PS it will reference a .js file.
To test this simply allow your authentication token to expire without a login. The error message in the .json file it sends you to indicates a failure in a Javascript file.
The release schedule for WIndows Azure PowerShell and the command line tools are both approximately once every 3 weeks. You can get the latest from the github repo at any time (as shown above).
I want to run PS scripts in C# exactly along the lines of Sample
This works on a Cloud Service (Web Role), on a standard server (with execution policy disabled), but not in a Azure Web Site.
Anyone know why this is so?
I don't know of any official documentation on the issue but there are multiple answers on the related forums where Microsoft employees talk about the limitations of not being able/allowed to launch external processes in Windows Azure Web Sites. See
PHP - Warning: exec(): Unable to fork
PDF Generator failing on Azure Website but not development machine
Recommended HTML to PDF Generator that works on Azure Website Reserved mode with Full Trust
You are probably trying to do that with your powershell script.
You could verify if that is in fact your issue by catching and looking at the exception(s) which should relate to security policies/permissions.
What's the minimal guaranteed list of functionality a Windows Azure instance will provide?
For example, I want to use Powershell from inside my Azure role. Currently all Windows Azure instances do have Powershell. What if Microsoft suddenly decides to stop deploying Powershell by default?
Is there a list of guaranteed components and functionality that I may assume unconditionally available inside Windows Azure instances?
There is no guarantee with respect to features currently available in Windows Azure. Obviously, Microsoft being the late-comer to the cloud game and an underdog, will not likely be cutting features out but only adding features in, but there is certainly no guarantee.
We do know that Azure instances are running Win2008 and you can lock them down to a certain version of a patch so that nothing is removed.
Interesting post, not sure if MSFT have a complete list of what's on offer (I do remember this being talked about in the early days when it went to general release), be nice to see a list. (*cough mark russinovich!)
Re: Powershell, its not going anywhere fast, its a core tenant of the Windows o/s and I could bet my house on it.
Startup tasks are your ideal preference to ensure you have what you need in Azure, for instance I install the Java JVM on the box (VM role) during a startup task, it's not there naively but it is when my code runs!
http://www.davidaiken.com/2011/01/19/running-azure-startup-tasks-as-a-real-user/