Powershell: validate URL path, incorrect path vs non public path - powershell

I am trying to validate a URL that, if valid, I will use to download a file. But I want to validate the domain and the full path BEFORE doing the download, so I can provide a meaningful error log.
Given paths like the following
www.validDomain.com/downloads/validDownload.zip
www.validDomain.com/downloads/invalidDownload.zip
www.invalidDomain.com/downloads/validDownload.zip
I want to be able to report
invalid path: /downloads/invalidDownload.zip
Invalid domain: www.validDomain.com
I can use
$uri = [System.Uri]$path
if (Resolve-DnsName -Name:$uri.host) {}
to test the domain. I can then use
if (Invoke-Webrequest $uri.OriginalString -DisableKeepAlive -UseBasicParsing -Method:head) {}
to test the full path without actually doing the download, but the only error I get is The remote server returned an error: (403) Forbidden., both when the path doesn't exist and when the path does exist but it isn't a publicly available path.
Is there any way, on the PowerShell side, to differentiate the permissions issue vs the incorrect path issue?
Or do I have no choice but to provide a wishy washy error in the log? I know effetely the path does not exist to the user who doesn't have permissions, I am just hoping there is some client side way to differentiate. But I may be running into the limitations of how the web has been implemented.

Related

SolrCloud Configset API upload returns 500 "KeeperErrorCode = NoNode"

Situation
First of all I must mention that I'm using Solr 8.1.1 and am running the default "solr -e cloud" to do some testing. This is running on a Windows Azure VM. I'm trying to create a PowerShell script that will do some setup on the SolrCloud. The first step in this is uploading a custom Configset. I was using https://lucene.apache.org/solr/guide/8_1/configsets-api.html as guide and the PowerShell command if you take away all the parameters and such boils down to the following:
Invoke-WebRequest -Uri "http://localhost:8983/solr/admin/configs?action=UPLOAD&name=MyConfig" -Method Post -ContentType "application/octet-stream" -InFile "config.zip"
EDIT: For clarity the contents of the ZIP is as follows: https://imgur.com/a/OHR1bf1
Problem
When I run the above command however I'm met with the following error:
Invoke-WebRequest : { "responseHeader":{ "status":500, "QTime":11}, "error":{ "msg":"KeeperErrorCode = NoNode for /configs/MyConfig/lang/contractions_ca.txt", "trace":"org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /configs/MyConfig/lang/contractions_ca.txt\r\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:114)\r\n\tat
org.apache.zookeeper.KeeperException.create(KeeperException.java:54)\r\n\tat org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792)\r\n\tat
org.apache.solr.common.cloud.SolrZkClient.lambda$create$7(SolrZkClient.java:415)\r\n\tat org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71)\r\n\tat
org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:415)\r\n\tat org.apache.solr.handler.admin.ConfigSetsHandler.createZkNodeIfNotExistsAndSetData(ConfigSetsHandler.java:201)\r\n\tat
org.apache.solr.handler.admin.ConfigSetsHandler.handleConfigUploadRequest(ConfigSetsHandler.java:181)\r\n\tat org.apache.solr.handler.admin.ConfigSetsHandler.handleRequestBody(ConfigSetsHandler.java:111)\r\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\r\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:796)\r\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:762)\r\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:522)\r\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)\r\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)\r\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\r\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\r\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\r\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\r\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\r\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\r\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\r\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\r\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\r\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\r\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\r\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\r\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\r\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\r\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\r\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\r\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\r\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\r\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\r\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\r\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:502)\r\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\r\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\r\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\r\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\r\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\r\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\r\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\r\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\r\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)\r\n\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)\r\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\r\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\r\n\tat java.lang.Thread.run(Thread.java:748)\r\n",
"code":500}}
At line:1 char:1
Observations
When I first "failed" I had created a zip file from my config which contained an additional top level folder (ea instead of MyConfig/solrconfig.xml etc my ZIP was MyConfig/MyConfig/solrconfig.xml) and when I used this the command was run successful but the second command (creating a collection) would fail because it could not find solrconfig.xml. This tells me that the ZIP is correctly present in the request and Solr does seem capable of processing it but once I correct it to an actual configset it massively fails?
EDIT: I was asked about this and whether using "conf" in the zip would work. As I mentioned here this results in a successful upload (https://imgur.com/a/JHLZ8td) however as you can see it does not match the other config sets and when you try to create a collection with this set you will get Error CREATEing SolrCore 'Test_shard1_replica_n1': Unable to create core [Test_shard1_replica_n1] Caused by: Can't find resource 'solrconfig.xml' in classpath or '/configs/Sitecore', cwd=C:\solr-8.1.1\server
Question(s)
What am I doing wrong? Is this a bug?
Going back through some work I did on SolrCloud a while ago, I am reminded of one annoying issue I hit:
I got odd issues uploading the schema config zip files if I had created that zip using "Send to Compressed Folder" in the Windows UI, or via Compress-Archive in PowerShell. I found that compressing the data with 7Zip did work, however.
I suspect there's something incompatible between the Windows zip code (which I think is quite old, and something they licensed ages ago?) and how Solr/ZooKeeper deals with extracting the files again?
I just ran into the same issue without using Windows zip code. I was trying to upload a configset to Solr 7.7.3 from a conf directory containing a "lang" subdirectory with a bunch of files. I got the NoNode error for /configs/_myconfigsetname_/lang/stopwords_eu.txt. The configset was being zipped on the fly through a recursive directory walk in Java, sending each filename to the Zip file using Java's ZipOutputStream. The resulting zipped bytes were then sent to Solr/Zookeeper.
This code worked fine for conf directories without subdirectories. It turned out that when there is a subdirectory, it was necessary to create a ZipEntry for the directory (e.g. lang/) before adding files to the Zip stream such as lang/stopwords_eu.txt.

citrus waitFor().file fails to read a file

I’m trying to use waitFor() in my Citrustest to wait for an output file on disk to be written by the process I’m testing. I’ve used this code
outputFile = new File “/esbfiles/blesbt/bl03orders.99160221.14289.xml");
waitFor().file(outputFile).seconds(65L).interval(1000L);
after a few seconds, the file appears in the folder as expected. The user I’m running the test code as has permissions to read the file. The waitFor(), however, ends in a timeout.
09:46:44 09:46:44,818 DEBUG dition.FileCondition| Checking file path '/esbfiles/blesbt/bl03orders.99160221.14289.xml'
09:46:44 09:46:44,818 WARN dition.FileCondition| Failed to access file resource 'class path resource [esbfiles/blesbt/bl03orders.99160221.14289.xml] cannot be resolved to URL because it does not exist'
What could be the problem? Can’t I check for files outside the classpath?
This is actually a bug in Citrus. Citrus is working with the file path instead of the file object and in combination with Spring's PathMatchingResourcePatternResolver this causes Citrus to search for a classpath resource instead of using the absolute file path as external file system resource.
You can fix this by providing the absolute file path instead of the file object like this:
waitFor().file(“file:/esbfiles/blesbt/bl03orders.99160221.14289.xml")
.seconds(65L)
.interval(1000L);
Issue regarding broken file object conversion has been opened: https://github.com/christophd/citrus/issues/303
Thanks for pointing to it!

MSDeploy reporting "could not find file" when using setParamFile

I'm using MSDeploy as part of a custom task in TFS2017 (Update 1) but I can't get the setParamFile option to work. It throws one of two errors. If I specify no path, it says "could not find file" (see trace below). If I specify the path to the file on the build machine, it says invalid path.
I'm sure it's something incredibly simply but it's got me beat and Google is, for once, not being my friend! Any help is greatly appreciated.
I have provided the log unaltered as all this is only on internal servers. Without the setParamFile parameter, it deploys fine without updating the variable connection strings I'm trying to update, obviously.
2017-04-15T22:41:13.5065833Z Executing the powershell script: D:\BuildAgent\tasks\MSDeployPackageMultiSync\1.1.16\MSDeployPackageSync.ps1
2017-04-15T22:41:13.6472111Z packageFile= Find-Files -SearchPattern D:\BuildAgent\_work\194fdc7b9\Data Collection App - Release\drop\_PublishedWebsites\Cabi.FormsAdmin.Api_Package\DCA_API.zip
2017-04-15T22:41:13.6472111Z packageFile= D:\BuildAgent\_work\194fdc7b9\Data Collection App - Release\drop\_PublishedWebsites\Cabi.FormsAdmin.Api_Package\DCA_API.zip
2017-04-15T22:41:13.6784674Z DCA_API.SetParameters.xml
2017-04-15T22:41:13.7097114Z Package= D:\BuildAgent\_work\194fdc7b9\Data Collection App - Release\drop\_PublishedWebsites\Cabi.FormsAdmin.Api_Package\DCA_API.zip
2017-04-15T22:41:13.7097114Z ParamFile= -setParamFile:'DCA_API.SetParameters.xml'
2017-04-15T22:41:13.7097114Z DestinationProvider= auto
2017-04-15T22:41:13.7097114Z DestinationComputer= prerelweb3,prerelweb4
2017-04-15T22:41:13.7097114Z Username=
2017-04-15T22:41:13.7097114Z AdditionalArguments=
2017-04-15T22:41:13.7253453Z Deploying package to prerelweb3,prerelweb4
2017-04-15T22:41:13.7253453Z Deploying to prerelweb3
2017-04-15T22:41:13.7253453Z "C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -source:package='D:\BuildAgent\_work\194fdc7b9\Data Collection App - Release\drop\_PublishedWebsites\Cabi.FormsAdmin.Api_Package\DCA_API.zip' -setParamFile:'DCA_API.SetParameters.xml' -dest:auto,computerName='prerelweb3',userName='',password='',authType='ntlm',includeAcls='False' -setParam:name='IIS Web Application Name',value='WebServices (Secure)\FormsAdmin' -setParam:kind=DestinationVirtualDirectory,scope='WebServices (Secure)\\FormsAdmin',value='e:\cabi_sites\WebServices_Secure\FormsAdmin' -allowUntrusted
**2017-04-15T22:41:14.1315861Z ##[error]System.Management.Automation.RemoteException: Error: Could not find file ''DCA_API.SetParameters.xml''.**
2017-04-15T22:41:14.1315861Z ##[error]System.Management.Automation.RemoteException: Error count: 1.
2017-04-15T22:41:14.1315861Z Deployment to prerelweb3 complete
2017-04-15T22:41:14.1315861Z Deploying to prerelweb4
2017-04-15T22:41:14.1315861Z "C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -source:package='D:\BuildAgent\_work\194fdc7b9\Data Collection App - Release\drop\_PublishedWebsites\Cabi.FormsAdmin.Api_Package\DCA_API.zip' -setParamFile:'DCA_API.SetParameters.xml' -dest:auto,computerName='prerelweb4',userName='',password='',authType='ntlm',includeAcls='False' -setParam:name='IIS Web Application Name',value='WebServices (Secure)\FormsAdmin' -setParam:kind=DestinationVirtualDirectory,scope='WebServices (Secure)\\FormsAdmin',value='e:\cabi_sites\WebServices_Secure\FormsAdmin' -allowUntrusted
**2017-04-15T22:41:14.4909648Z ##[error]System.Management.Automation.RemoteException: Error: Could not find file ''DCA_API.SetParameters.xml''.**
2017-04-15T22:41:14.4909648Z ##[error]System.Management.Automation.RemoteException: Error count: 1.
2017-04-15T22:41:14.4909648Z Deployment to prerelweb4 complete
I assume that you are using this task: MSDeployAllTheThings. According to the logs, you are using it from Release. So you need to make sure that the "DCA_API.SetParameters.xml" file is published to artifact drop during the build and is also downloaded correctly during release. And then, you need to enter the absolute path in the Additional Arguments so that the task can find the file. For your scenario, if the "DCA_API.SetParameters.xml" file is placed in the same folder with "DCA_API.zip" file, you need to use this:
-setParamFile:"D:\BuildAgent\_work\194fdc7b9\Data Collection App - Release\drop\_PublishedWebsites\Cabi.FormsAdmin.Api_Package\DCA_API.SetParameters.xml"
I tried with absolute path at my side and didn't see any issue with it.

Use AppCmd to LIST CONFIG in APPHOST only

I have a requirement to use powershell to configure IIS7.5 on WebApplications that have not yet had code deployed (possibly at all, possibly old/broken web.configs exist) to the file system. I would like to be able to do this all at the APPHOST level. (Note at the bottom about use of Powershell > AppCmd).
I can SET all the values properly, however, being somewhat diligent, I like to also validate the values were set properly by retrieving them after setting.
Here's the scenario:
I can set this value using AppCmd so the setting is applied at the APPHOST level using the /Commit:APPHOST flag. However, I havent found a way to READ the values exclusively at the APPHOST level.
Setting the Code is successful:
C:\Windows\System32\inetsrv\appcmd.exe set config "webSiteName/webAppName" -section:system.webServer/security/authentication/anonymousAuthentication /enabled:"True" /commit:apphost
However, I cant find a way to read the values using AppCmd (or Powershell):
Running the following AppCmd returns an error due to the broken pre-existing web.config in the folder (the specific error is unimportant, as it is reading the WebApp's web.config instead of the ApplicationHost.config/APPHOST):
C:\Windows\System32\inetsrv\appcmd.exe list config "MACHINE/WEBROOT/APPHOST/webSiteName/webAppName" -section:system.webServer/security/authentication/anonymousAuthentication
ERROR ( message:Configuration error
Filename: \\?\c:\inetpub\wwwroot\webSiteName\webAppName\web.config
Line Number: 254
Description: The configuration section 'system.runtime.caching' cannot be read because it is missing a section declaration
. )
Note: I would prefer to do this all in Powershell instead of using AppCmd, so if anyone has the syntax for modifying the APPHOST settings for anonymousAuthentication section of a WebApplication, that lives under a Website, from inside Powershell (Get-WebConfiguration seems to only use the WebApp web.config), that would be totally awesome and much appreciated!
Here's how to do this in PowerShell:
[Reflection.Assembly]::Load(
"Microsoft.Web.Administration, Version=7.0.0.0,
Culture=Neutral, PublicKeyToken=31bf3856ad364e35") > $null
$serverManager = New-Object Microsoft.Web.Administration.ServerManager
$config = $serverManager.GetApplicationHostConfiguration()
$anonymousAuthenticationSection = $config.GetSection("system.webServer/security/authentication/anonymousAuthentication", "simpleasp.net")
Write-Host "Current value: " $anonymousAuthenticationSection["enabled"]
# Now set new value
$anonymousAuthenticationSection["enabled"] = $true
$serverManager.CommitChanges()

Breaking MsBuild package & deploy into separate MsBuild and MsDeploy commands

I'm having a few problems breaking out an MsBuild package+deploy command into two separate commands. (I need to do this to pass additional parameters to MsDeploy).
The command that works fine looks like this:
msbuild "src\Solution.sln"
/P:Configuration=Deploy-Staging
/P:DeployOnBuild=True
/P:DeployTarget=MSDeployPublish
/P:MsDeployServiceUrl=https://192.168.0.1:8172/MsDeploy.axd
/P:DeployIISAppPath=staging.website.com
/P:AllowUntrustedCertificate=True
/P:MSDeployPublishMethod=WmSvc
/P:CreatePackageOnPublish=True
/P:UserName=staging-deploy
/P:Password=xyz
The separated packaging command looks like this:
msbuild "src\Solution.sln"
/P:Configuration=Deploy-Staging
/P:DeployOnBuild=True
/P:DeployTarget=Package
/P:_PackageTempDir=C:\temp\web
which works fine. But then the MsDeploy portion:
msdeploy
-verb:sync
-allowUntrusted
-usechecksum
-source:manifest=
'src\WebProject\obj\Deploy-Staging\Package\WebProject.SourceManifest.xml'
-dest:auto,ComputerName=
'https://192.168.0.1:8172/MsDeploy.axd?site=staging.website.com',
username='staging-deploy',password='xyz',authType='basic',includeAcls='false'
-enableRule:DoNotDeleteRule
fails, with the following error in WmSvc.log
wmsvc.exe Error: 0 : Attempted to perform an unauthorized operation.
setAcl/C:\temp\web (Read)
ProcessId=15784
ThreadId=31
DateTime=2011-03-30T14:57:02.4867689Z
Timestamp=3802908721815
wmsvc.exe Error: 0 : Not authorized.
Details: No rule was found that could authorize user 'staging-deploy',
provider 'setAcl', operation 'Read', path 'C:\temp\web'.
(and several more Read/Write operations)
Something is clearly going wrong with the paths it's trying to access (as it works fine with the other method) - I'm not sure it's even trying to use the iisApp targeting correctly, and at the moment I don't think the correct web.config's will be deployed either.
I've got this fixed now - I needed a different command to the one the automatically generated .cmd file was using, but comparing the two allowed me to fix it up (thanks #Vishal R. Joshi)
The differences I needed was:
basic authentication
allow untrusted certificates
?site=staging.webserver on the end of the MsBuild.axd path, as with my original command
override the IIS Web App name that is set in the params file
enable the do not delete rule
The winning command is as follows:
msdeploy
-verb:sync
-allowUntrusted
-source:package='src\WebProject\obj\Deploy-Staging\Package\WebProject.zip'
-dest:auto,ComputerName=
'https://192.168.0.1:8172/MsDeploy.axd?site=staging.website.com',
username='staging-deploy',password='xyz',authType='basic',includeAcls='false'
setParamFile:
"src\WebProject\obj\Deploy-Staging\Package\WebProject.SetParameters.xml"
-setParam:name='IIS Web Application Name',value='staging.website.com'
-enableRule:DoNotDeleteRule
-disableLink:AppPoolExtension -disableLink:ContentExtension
-disableLink:CertificateExtension
Hope this helps someone!
Add a delegation rule on the server using inetmgr to allow staging-deploy to carry out set-Acl operations.
Inetmgr -> Click on server node -> Management Service Delegation (in Management) -> Click Add rule to the right -> Choose the template labelled "Set Permissions for Applications" -> Accept defaults and click OK.
This should let you deploy any package or manifest with setAcl as long as the user you are deploying as, has permissions to the site you are deploying to.
You are able to specify the -setParam:name='',value='' flag when calling the MyProject.deploy.cmd file that is created when you generate a Package from a web project. The cmd is a friendly wrapper around msdeploy.exe, so you have no need to specify all the rest of the defaults.
Here's the details: http://evolutionarydeveloper.blogspot.co.uk/2013/05/specifying-environment-variables-at.html