i'm getting a limitation with cmdlet of Azure Datalake gen2:
https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-powershell
I'm using this cmdlet:
Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem $filesystemName -Path $dirname -Recurse -FetchProperty
to get all file and folder ACL from root but it has 5000 object limitation and it show this message when i run it to a folder with more than 5000 object:
basically, with that token i can continue from the last extracted(manually is crazy to do becouse maybe we have million file on datalake).
Is possible to avoid it or to loop in some way?
Here the script that i'm using(it works fine, i don't report all file ma only folder from root):
$dir = Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem "datalake" -Recurse -FetchProperty
$FileOutdtk = "C:\Temp\file.csv"
Clear-Content $FileOutdtk
Add-Content $FileOutdtk ('"Path"^"IsDirectory"^"Owner"^"DisplayName Owner"^"Owner Permissions"^"Group"^"DefaultScope"^"AccessControlType"^"EntityId"^"DisplayName Gruppo"^"PermissionsACL"')
foreach ($directory in $dir) {
if($directory.IsDirectory -eq $true){
if($directory.Owner -imatch "superuser"){
foreach ($ACLs in $directory.ACL){
if($ACLs.EntityId -eq $null ){
Add-Content $FileOutdtk ('"' + $directory.Path + '^' + $directory.IsDirectory + '^' + $directory.Owner + '^' + "" + '^' + $directory.Permissions.Owner + '^' + $directory.Group + '^' + $ACLs.DefaultScope + '^' + $ACLs.accesscontroltype + '^' + $ACLs.EntityId + '^' + "" + '^' + $ACLs.Permissions + '"')
}
else{
$GruppiEntityId = Get-AzureADGroup -ObjectId $ACLs.EntityId
Add-Content $FileOutdtk ('"' + $directory.Path + '^' + $directory.IsDirectory + '^' + $directory.Owner + '^' + "" + '^' + $directory.Permissions.Owner + '^' + $directory.Group + '^' + $ACLs.DefaultScope + '^' + $ACLs.accesscontroltype + '^' + $ACLs.EntityId + '^' + $GruppiEntityId.displayname + '^' + $ACLs.Permissions + '"')
}
}
How i can loop that cmdlet to get up to 5000 object?
Thanks a lot
If you want to list all items in one Azure data lake gen2 folder, please refer to the following script
$storageAccount = Get-AzStorageAccount -ResourceGroupName "<>" -AccountName "<>"
$ctx = $storageAccount.Context
$fileSystem="test"
$dirName="testFolder"
$Token = $Null
$Max=2000
do{
$items=Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem $fileSystem -Path $dirName -Recurse -FetchProperty -ContinuationToken $Token -MaxCount $Max
$items
if($items.Length -le 0) { Break;}
$Token = $items[$items.Count -1].ContinuationToken;
}
While ($Token -ne $Null)
Related
I am a beginner at powershell script and I am stuck
I am trying to upload 2 xml files with the same name to FTP, but I am getting the following error
Exception calling "UploadFile" with "2" argument(s): "An
exception occurred during a WebClient request." At
C:\powershell\ExternalWidgets\upload-to-ftp.ps1:51 char:16
+ $webclient.UploadFile($uri.AbsoluteUri, $LocalPath)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : WebException`
Here is the script:
#SET CREDENTIALS
$credentials = new-object System.Net.NetworkCredential($user, $pass)
cd $target
$DirectoryContent = Get-ChildItem -Directory
foreach($item in $DirectoryContent){
cd $item
$currentFolderItems = Get-ChildItem
foreach( $subfolderItem in $currentFolderItems)
{
if ($subfolderItem.name -eq "package.json") {
$subItem = Get-ChildItem -Directory
foreach($subItem in $subItem)
{
if ($subItem.name -eq "coverage") {
$files = Get-ChildItem -Path $subItem
$n = 0
foreach ($file in $files)
{
#rename coverage file
if($file.Name -eq "cobertura-coverage.xml") {
$newFileName=$file.Name.Replace("cobertura-coverage.xml","cobertura-coverage$n.xml")
Rename-Item $file -NewName $newFileName
$newFileName
}
$n++
}
}
}
$webclient = New-Object System.Net.WebClient
$webclient.Credentials = New-Object
System.Net.NetworkCredential($user,$pass)
$uri = New-Object System.Uri($ftp + "/" + $folder)
#Delete files from FTP
# $fileUploaded =($uri + "\" + $newFileName)
# $fileUploaded
# Remove-Item $fileUploaded
$LocalPath =($target + "\" + $item + "\" + "coverage" + "\" +
$newFileName).Replace(".\", "").Replace('\', '/')
$webclient.UploadFile($uri, $LocalPath)
#cd..
}
}
cd ..
}
I also get an error when I want to rename one of the files. What step am I missing? Thank you
How do I check if the copy was successful before the script goes to the next command?
$src_dir_local = 'D:\temp\tobemoved\'
$dst_dir_local = 'D:\temp\moved\'
$log_dir = 'D:\temp\log\'
$log_file = 'backup.log'
$ofs_char = "`n"
$curr_date = Get-Date -UFormat '%Y-%m-%d'
$curr_date_time = Get-Date
$s3_bucket = 's3://mybucket/'
$s3_storage_class = 'STANDARD_IA'
$files = Get-ChildItem $src_dir_local
write-output $src_dir_local + $file + $curr_date
write-output $s3_command
aws s3 ls
write-output 'looping through files....'
foreach($f in $files){
$outputfilename = $f.Name
$s3_move_command = 'time aws s3 cp --quiet' + ($src_dir_local + $outputfilename + ' ' + $s3_bucket + $curr_date + '/' + $outputfilename + ' --storage-class ' + $s3_storage_class)
write-output $outputfile
write-output 'Start process'
Invoke-Expression $s3_move_command
move-item -path ($src_dir_local + $outputfilename) -destination ($dst_dir_local + '_mvd_' + $outputfilename) -whatif
write-output 'End process'
write-output 'Start moving process'
$log_message = ($ofs_char + 'File moved ' + $outputfile + ' ' + $((Get-Date).ToString()))
if(Test-Path ($log_dir + $log_file)){
Add-Content ($log_dir + $log_file) $log_message
}else{
New-Item -path $log_dir -name $log_file -type 'file' -value 'Initial log '
Add-Content ($log_dir + $log_file) $log_message
}
}
In general, you would write:
aws s3 cp $src $dst
If ($lastexitcode -eq 0) {
...
} Else {
...
}
See the docs here.
If you must use Invoke-Expression, then see the linked Stack Overflow answer.
I have this script that creates files using bcp. As BCP doesn't export data with header i need to join both files into. If i run the code in another script it creates the csv file but if i run this script it fails saying files DOES NOT EXIST, that is the last section of the script below (#THE PROBLEM STARTS HERE).
Appreciate any help.
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$PSDefaultParameterValues['*:ErrorAction']='Stop'
$bcp = 'c:\Program Files\Microsoft SQL Server\100\Tools\Binn\bcp.exe'
$delimiter = '","'
$server = 'BLABLABLA'
$database = 'DMB'
$schema = '.dbo.'
$dbschema = $database + $schema
$atresult = "#result"
$commas = " + ',','') + "
$head1 = 'Use '
$head2 = ' Declare ' # atresult
$head3 = ' varchar(max) select ' # atresult
$head4 = ' = COALESCE(' #atresult $commas
$head5 = 'COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = ' # tablename
$head6 = ' select ' #atresult
$tables = #('EntityTypes_20181205','FieldFieldTypeInfo')
$output_path = 'D:\Temp\DataModelBuild\'
# Deletes any file in Destination Folder
#if ( (Get-ChildItem $output_path | Measure-Object).Count -ne 0) {}
Get-ChildItem -Path $output_path -Include *.* -File -Recurse | foreach { $_.Delete()}
foreach ($element in $tables)
{
# GET COLUMN NAMES into header file
$outputFile = $output_path + $element + ".header"
$NQ = '"' + $head1 + $database + $head2 + $atresult + $head3 + $atresult + $head4 + $atresult + $commas + $head5 + ''' + $element + ''' + $head6 + $atresult + '"'
#$arglist = "(" + $NQ + " queryout " + $outputFile + ", -S " + $server + " -T -t" + $delimiter + " -c -k)"
$arglist = #('"',
$head1,
$database,
$head2,
$atresult,
$head3,
$atresult,
$head4,
$atresult,
$commas,
$head5,
"'$element'",
$head6,
$atresult,
'"',
" queryout $outputFile",
"-S $server -T -t$delimiter -c -k")
#write-output $arglist
Start-Process -FilePath $bcp -ArgumentList $arglist
#Read-Host
# GET DATA into data file
$outElement = $dbschema + "[" + $element + "]"
$outputFile = $output_path + $element + ".data"
$arglist = #($outElement, "out $outputFile", "-S $server -T -t$delimiter -c -k")
Start-Process -FilePath $bcp -ArgumentList $arglist
#write-output $arglist
#Read-Host
# Merge header and data to csv
#Invoke-Expression "&'$ExtPs1' -FilePath $output_path -FileName $element"
#Write-Output "call "
#Invoke-Expression "&'D:\idna\PedroTemp\cp2csv.ps1' $output_path $element"
#Read-Host
#THE PROBLEM STARTS HERE
$File1 = #("$output_path$element.header")
$File2 = #("$output_path$element.data")
$File3 = #("$output_path$element.csv")
write-output $File1, $File2, $File3
If (-not (Test-Path -Path $File1)) {
Write-Error "File DOES NOT EXIST '$File1'" -ErrorAction Stop
} else {
Write-Host "File EXIST"
}
Add-Content $File3 -value (Get-Content $File1)
Add-Content $File3 -value (Get-Content $File2)
#write-output $File1, $File2, $File3 -ErrorAction Stop
Read-Host
}
My question is why isn't the script detecting the file?
Is there any problem with the string?
The strings for the pathes aren't build correctly. When you use a member in a string you have to enclose the variable and the member with $().
$File1 = "$output_path$($element.header)"
I am running a script which has $FileName variable containing the absolute path with spaces. Due to the space within the directory name and file name, the script fails to executes without finding the actual path. All I need is to add $FilePath within double quotes. How should I append double quotes in the beginning and end of a string?
For example
"X:\Movies\File One\File One.txt"
Script:
$FilePath = Join-Path $Path $($Dir + "\" + $File + “.txt”)
$FilePath
Current OutPut:
X:\Movies\File One\File One.txt
In addition to the backtick escape character (`), you can use the -f format operator:
$FilePath = Join-Path $Dir -ChildPath "$File.txt"
$FilePathWithQuotes = '"{0}"' -f $FilePath
This will ensure that $FilePath is expanded before being placed in the string
$FilePath = Join-Path $Path $($Dir + "\" + $File + “.txt”)
"`"$FilePath`""
...would output...
"X:\Movies\File One\File One.txt"
This is an example of variable expansion in strings.
Of course, if the path you want to quote could contain " quotes itself, for example in a future "powershell for linux", you'd need to escape the " in a context specific way.
Any one of these should work:
$FilePath1 = """" + (Join-Path $Path $($Dir + "\" + $File + ".txt")) + """"
$FilePath2 = "`"" + (Join-Path $Path $($Dir + "\" + $File + ".txt")) + "`""
$FilePath3 = '"{0}"' -f (Join-Path $Path $($Dir + "\" + $File + ".txt"))
$FilePath4 = '"' + (Join-Path $Path $($Dir + "\" + $File + ".txt")) + '"'
$FilePath5 = [char]34 + (Join-Path $Path $($Dir + "\" + $File + ".txt")) + [char]34
fastest solution (but a bit ugly) to add quotes around any string:
$dir = "c:\temp"
$file = "myfile"
$filepath = [string]::Join("", """", $dir,"\", $file, ".txt", """")
I have a script that obtains that gathers information about a computer then outputs it to a csv file. You can see I have tried multiple approaches to this issue and cannot quite get any of them to work. A fix to this, or a completely different solution would be welcome.
#Objective is to create a csv file with
# "hostname of public IP","public IP","machine name",date,time-24hour
# for example: (note no header in file)
# "97-94-177-139.dhcp.ftwo.tx.charter.com","97.94.177.139","IT-BHOLLING",8/23/2014,06:52:35
# quotes around text fields are an optional objective (helps with some csv import engines)
# some approaches require that the path c:\IP\Working exists (or change lines 46 & 47
# Change line 61 to try each output method
#Variables
# I am defining website url in a variable
$url = "http://checkip.dyndns.com"
# Creating a new .Net Object names a System.Net.Webclient
$webclient = New-Object System.Net.WebClient
# In this new webdownlader object we are telling $webclient to download the url $url
#get public IP address
$Ip = $webclient.DownloadString($url)
# Just a simple text manuplation to get the ipadress form downloaded URL
# If you want to know what it contain try to see the variable $Ip
#$Ip2 = $Ip.ToString()
#$ip3 = $Ip2.Split(" ")
#$ip4 = $ip3[5]
#$ip5 = $ip4.replace("</body>","")
#-- or - do this
$FinalIPAddress = $Ip.ToString().Split(" ")[5].replace("</body>","").replace("</html>","")
#get machine name
$MNme = $env:computername
#Get Hostname from IP address
try {
$Resolved = [system.net.dns]::GetHostEntry($FinalIPAddress)
}
catch {
$Resolved = ((&nslookup $FinalIPAddress)|where {$_ -match "^Name:"}).split(':')[1].trim()
}
#$Qstring = "select * from win32_pingstatus where address=" + """$FinalIPAddress""" + " AND ResolveAddressNames=True"
#$Qstring = "'" + $Qstring + "'"
# $AName = Get-WmiObject -Query $Qstring |
# select ProtocolAddressResolved | ConvertTo-Csv
#$AName = Get-WMIObject -q 'select * from win32_pingstatus where address="97.93.177.138" AND ResolveAddressNames=True'|
# select ProtocolAddressResolved | ConvertTo-Csv
#$AName[2..2].ToString()
$FinalIPAddress = $FinalIPAddress.Replace("`r`n","")
#get date
$GDte = Get-Date -format("G")
$GTme = Get-Date -format("u")
$infile = "C:\IP\Working\" + $MNme + ".txt"
$outfile = "C:\IP\Working\" + $MNme + ".csv"
$Dsplit = $GDte.split(" ")
$Dte = $Dsplit[0]
$Tsplit = $GTme.split(" ")
$Tme = $Tsplit[1].replace("Z","")
#remove previous files
If (Test-Path $infile){
Remove-Item $infile
}
If (Test-Path $outfile){
Remove-Item $outfile
}
#Write accumulated data to a file; modify $CMethod to test each alternate approach to creating csv file
$CMethod = "D"
if ($CMethod -eq "A") {
#this method produces a file that works with Access but not with Excel. Excel sees it as unicode-text
$content = """$Resolved""" + "," + """$FinalIPAddress""" + "," + """$MNme""" + "," + $Dte + "," + $Tme
$content > $outfile
}
if ($CMethod -eq "B") {
#This method produces an empty csv file, with a few tweaks it procudes a file with Length info instead of results
#tried with quotes
$content = """$Resolved""" + " " + """$FinalIPAddress""" + " " + """$MNme""" + " " + $Dte + " " + $Tme
#also tried without quotes
#$content = $Resolved + " " + $FinalIPAddress + " " + $MNme + " " + $Dte + " " + $Tme
$content > $infile
import-csv $infile -delimiter " " | export-csv -NoTypeInformation $outfile
}
If ($CMethod -eq "C") {
$content = """$Resolved""" + " " + """$FinalIPAddress""" + " " + """$MNme""" + " " + $Dte + " " + $Tme
add-content $infile $content
import-csv $infile -delimiter " " | export-csv -NoTypeInformation $outfile
}
if ($CMethod -eq "D") {
#Yet another method, convert to object first
#$content = """$Resolved""" + " " + """$FinalIPAddress""" + " " + """$MNme""" + " " + $Dte + " " + $Tme
#$content = """$Resolved""" + "," + """$FinalIPAddress""" + "," + """$MNme""" + "," + $Dte + "," + $Tme
$content = $Resolved + "," + $FinalIPAddress + "," + $MNme + "," + $Dte + "," + $Tme
$psObject = $null
$psObject = New-Object psobject
$Csplit = $content.Split(",")
#alternate approaches to converting PSobjecdt to csv
foreach($o in $Csplit)
{
Add-Member -InputObject $psobject -MemberType noteproperty -Name $o -Value $o -PassThru
}
$psObject | Export-Csv $outfile -NoTypeInformation
#Add-Member -InputObject $psobject -MemberType noteproperty -Name $Csplit -Value $Csplit
#$psObject | Export-Csv $outfile -NoTypeInformation
}
#echo to console
"Mehtod Choosen = " + $CMethod
$content
$Csplit
"end of the script....."
You can use Add-Content -Path "FilePath" -Value "$Resolved,$FinalIPAddress,$MNme,$Dte,$Tme".
Then you have no header and the data seperated with ",".
I would suggest letting PowerShell do the CSV creation for you. All that you need to do for that is to create a custom object with the properties you're after, then you can use the Export-Csv or ConvertTo-Csv cmdlets. You can also control the encoding used since it sounds like you might be having some encoding issues with the programs that are consuming your CSV. I've modified your code below to create a PSObject. See if this works for you:
$url = "http://checkip.dyndns.com"
$Now = Get-Date
$webclient = New-Object System.Net.WebClient
$ErrorString = "<ERROR>"
$CsvEncoding = "ASCII" # Any encoding from [System.Text.Encoding] should work
$outfile = "C:\IP\Working\{0}.csv" -f $env:computername
If (Test-Path $outfile){
Remove-Item $outfile
}
try {
$Ip = $null
$Ip = $webclient.DownloadString($url)
}
catch {
Write-Warning ("Error getting IP address: {0}" -f $_.Exception.Message)
}
if ($Ip -match ".*IP Address:\s*((\d{1,3}\.){3}\d{1,3}).*") {
$FinalIpAddress = $matches[1]
#Get Hostname from IP address
try {
$Resolved = [system.net.dns]::GetHostEntry($FinalIpAddress) | select -ExpandProperty HostName
}
catch {
try {
$Resolved = ((&nslookup $FinalIpAddress)|where {$_ -match "^Name:"}).split(':')[1].trim()
}
catch {
$Resolved = $ErrorString
Write-Warning ("Error resolving IP address '$FinalIpAddress': {0}" -f $_.Exception.Message)
}
}
}
else {
$FinalIpAddress = $ErrorString
$Resolved = $ErrorString
}
# This won't work as is in PSv2, but a simple mod would fix it:
$ReturnObject = [PSCustomObject] #{
Resolved = $Resolved
FinalIpAddress = $FinalIpAddress
MNme = $env:computername
Dte = $Now.ToShortDateString()
Tme = $Now.ToString("HH:mm:ss")
}
# At this point, you can let PS do all of the work to create a CSV
# Create normal CSV with encoding defined above:
$ReturnObject | Export-Csv -Path $outfile -Encoding $CsvEncoding -NoTypeInformation
# Create a CSV w/o header (this line will actually append a second line to $outfile):
$ReturnObject | ConvertTo-Csv -NoTypeInformation | select -Skip 1 | Out-File -FilePath $outfile -Encoding $CsvEncoding -Append