Remove time and tag from fluentd output plugin stdout with json
Fluentd's output plugin produces output like:
2017-11-28 11:43:13.814351757 +0900 tag: {"field1":"value1","field2":"value2"}
So timestamp and tag are before the json. How can I remove these fields - I only like to have the json output
<match pattern>
#type stdout
</match>
expected output:
{"field1":"value1","field2":"value2"}
Set the json format type which by default doesn't includes time and tag in output:
<match pattern>
#type stdout
<format>
#type json
</format>
</match>
Did you try filters?
<filter pattern>
#type record_transformer
<record>
${tag}
</record>
</filter>
Related
Since I upgraded NLog to the latest version 5.1.0, logging special characters (e.g. Ö, ä or ß) stopped working. Outout to file is like: ö ä ü for these chars.
I checked the log file with NotePad++ and it says that the encoding is ANSI. However, I have configured NLog to use utf-8. My config:
<target name="logfile" xsi:type="File" fileName="${basedir}/Logs/nLog.csv" archiveAboveSize="50000000" archiveNumbering="Sequence" maxArchiveFiles="3" encoding="utf-8" keepFileOpen="true">
<layout xsi:type="CsvLayout">
<column name="time" layout="${longdate}" />
<column name="level" layout="${level}"/>
<column name="category" layout="${event-context:item=category}" />
<column name="message" layout="${message}" />
</layout>
</target>
What am I missing?
Based on the comments, I was able to solve the problem. However, I do not know what the real cause of the problem was.
Solution: remove encoding="utf-8" from the nLog config AND read the log file as Encoding.UTF-8 (i.e. when opening the file with a StreamReader).
I don't know why this works since nLog uses UTF-8 as default...
I have log lines in Kubernetes:
2022-06-12T19:55:19.014511382Z stdout F dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
2022-06-12T19:55:19.014515391Z stdout F Ending HttpMessageHandler cleanup cycle after 0.0016ms - processed: 0 items - remaining: 1 items
2022-06-12T19:55:29.010438412Z stdout F dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100]
2022-06-12T19:55:29.010512909Z stdout F Starting HttpMessageHandler cleanup cycle with 1 items
2022-06-12T19:55:29.010518914Z stdout F dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
2022-06-12T19:55:29.010532801Z stdout F Ending HttpMessageHandler cleanup cycle after 0.002ms - processed: 0 items - remaining: 1 items
I'm trying to parse these lines using the multiline plugin. I have the following config:
<source>
#type tail
#id in_tail_container_logs
path /var/log/containers/*namespace*.log
pos_file /var/log/fluentd-containers.log.pos
tag kubernetes.*
read_from_head true
<parse>
#type multiline
format_firstline /^(?<logtime>[^\]]*) \w+ \w (?<level>\w+): (?<classpath>(\w+\.?)+\[\d+\])/
format1 /^[^\]]* \w+ \w \s+ (?<message>.+)/
</parse>
</source>
<filter kubernetes.var.log.containers.**.log>
#type kubernetes_metadata
</filter>
<match kubernetes.var.log.containers.**namespace**.log>
#type loggly
loggly_url "https://logs-01.loggly.com/inputs/#{ENV['TOKEN']}/"
</match>
I can see the logs coming into Loggly, but they're missing all the fields I'm adding in the multiline parser's config. I have debug logging enabled and I can see that the loggly plugin is getting the records without the fields before sending them.
I tested the regular expressions against the lines individually using fluentular and they seem to work individually. Is there something I'm missing from my config that could be causing the fields to be missing?
I would like to comment out an XML node in a configuration file using Powershell 3.0.
For instance, if this is my config.xml file:
<node>
<foo type="bar" />
</node>
I would like my script to change the file to this:
<node>
<!-- <foo type="bar" /> -->
</node>
I am looking to do this using Powershell 3.0's native XML/XPATH functionality, and not match/regex-based string replacement.
Use CreateComment() to create a new comment node containing the existing node's XML, then remove the existing:
$xml = [xml]#'
<node>
<foo type="bar" />
</node>
'#
# Find all <foo> nodes with type="bar"
foreach($node in $xml.SelectNodes('//foo[#type="bar"]')){
# Create new comment node
$newComment = $xml.CreateComment($node.OuterXml)
# Add as sibling to existing node
$node.ParentNode.InsertBefore($newComment, $node) |Out-Null
# Remove existing node
$node.ParentNode.RemoveChild($node) |Out-Null
}
# Export/save $xml
I'm trying to use web.config transformation in my release pipeline. However, no matter what I'm doing, I always get
2019-11-11T12:19:46.1172474Z ##[warning]Unable to apply transformation for the given package. Verify the following.
I have a Web.config and a Web.Elastic.config. The project has no dependentUpon for any .config files in it's csproj, and Web.Elastic.config has content as build action and is in the zip file generated from the build task.
In addition, I disabled config transformations during building just to be sure. I'm not sure what else I can do. This happens both when using the File Transform Task Preview as well as the XML transformation option during the IIS Web App Deploy task.
The File Transform task is configured like this:
I reduced my configs to this to see if there's something wrong with the transformation itself:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<configSections>
<section name="apiConfig" type="System.Configuration.NameValueSectionHandler, System, Version=1.0.5000.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
</configSections>
<apiConfig>
<add key="ClientBasetUrl" value="http://localhost:4200" />
</apiConfig>
<system.web>
<compilation debug="true" targetFramework="4.6.2">
<assemblies>
<add assembly="System.Net.Http.WebRequest, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
</assemblies>
</compilation>
<!-- This will handle requests up to 20MB -->
<httpRuntime targetFramework="4.6.1" maxRequestLength="20480" />
</system.web>
</configuration>
with the transformation looking like this:
<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<apiConfig xdt:Transform="Replace">
<add key="ClientBasetUrl" value="https://elastic.OURPROJECT.com" />
</apiConfig>
<system.web xdt:Transform="Replace">
<customErrors mode="Off" />
<compilation debug="true" targetFramework="4.6.2" />
<httpRuntime targetFramework="4.6.1" maxRequestLength="20480" />
</system.web>
</configuration>
With System.Debug set to true, the File Transformation task provides the following logs, which are not very helpful:
2019-11-11T12:19:43.1224281Z ##[debug]agent.TempDirectory=C:\vstsagent\A1\_work\_temp
2019-11-11T12:19:43.1289319Z ##[debug]loading inputs and endpoints
2019-11-11T12:19:43.1292645Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
2019-11-11T12:19:43.1301246Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
2019-11-11T12:19:43.1304962Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
2019-11-11T12:19:43.1307442Z ##[debug]loading INPUT_ENABLEXMLTRANSFORM
2019-11-11T12:19:43.1309334Z ##[debug]loading INPUT_FILETYPE
2019-11-11T12:19:43.1316311Z ##[debug]loading INPUT_FOLDERPATH
2019-11-11T12:19:43.1316632Z ##[debug]loading INPUT_XMLTRANSFORMATIONRULES
2019-11-11T12:19:43.1322390Z ##[debug]loaded 7
2019-11-11T12:19:43.1334690Z ##[debug]Agent.ProxyUrl=undefined
2019-11-11T12:19:43.1336439Z ##[debug]Agent.CAInfo=undefined
2019-11-11T12:19:43.1336699Z ##[debug]Agent.ClientCert=undefined
2019-11-11T12:19:43.1336884Z ##[debug]Agent.SkipCertValidation=undefined
2019-11-11T12:19:43.2867243Z ##[debug]check path : C:\vstsagent\A1\_work\_tasks\FileTransform_8ce97e91-56cc-4743-bfab-9a9315be5f27\1.1.6\task.json
2019-11-11T12:19:43.2867893Z ##[debug]adding resource file: C:\vstsagent\A1\_work\_tasks\FileTransform_8ce97e91-56cc-4743-bfab-9a9315be5f27\1.1.6\task.json
2019-11-11T12:19:43.2868317Z ##[debug]system.culture=en-US
2019-11-11T12:19:43.2882674Z ##[debug]check path : C:\vstsagent\A1\_work\_tasks\FileTransform_8ce97e91-56cc-4743-bfab-9a9315be5f27\1.1.6\node_modules\webdeployment-common-v2\module.json
2019-11-11T12:19:43.2883957Z ##[debug]adding resource file: C:\vstsagent\A1\_work\_tasks\FileTransform_8ce97e91-56cc-4743-bfab-9a9315be5f27\1.1.6\node_modules\webdeployment-common-v2\module.json
2019-11-11T12:19:43.2884521Z ##[debug]system.culture=en-US
2019-11-11T12:19:43.2900694Z ##[debug]folderPath=C:\vstsagent\A1\_work\r2\a\SLX-Backend\drop\OURPROJECT.zip
2019-11-11T12:19:43.2902832Z ##[debug]Finding files matching input: C:\vstsagent\A1\_work\r2\a\SLX-Backend\drop\OURPROJECT.zip
2019-11-11T12:19:43.2907625Z ##[debug]fileType=xml
2019-11-11T12:19:43.2908713Z ##[debug]targetFiles=null
2019-11-11T12:19:43.2911002Z ##[debug]enableXmlTransform=true
2019-11-11T12:19:43.2912768Z ##[debug]xmlTransformationRules=-transform **\OURPROJECT\obj\Release\Package\PackageTmp\Web.Elastic.config -xml **\OURPROJECT\obj\Release\Package\PackageTmp\Web.config
2019-11-11T12:19:43.2916829Z ##[debug]This is zip package
2019-11-11T12:19:43.2919443Z ##[debug]Agent.TempDirectory=C:\vstsagent\A1\_work\_temp
2019-11-11T12:19:43.2919672Z ##[debug]Agent.TempDirectory=C:\vstsagent\A1\_work\_temp
2019-11-11T12:19:43.2927405Z ##[debug]extracting C:\vstsagent\A1\_work\r2\a\SLX-Backend\drop\OURPROJECT.zip to C:\vstsagent\A1\_work\_temp\temp_web_package_9699193652416063
2019-11-11T12:19:45.9624316Z ##[debug]extracted C:\vstsagent\A1\_work\r2\a\SLX-Backend\drop\OURPROJECT.zip to C:\vstsagent\A1\_work\_temp\temp_web_package_9699193652416063 Successfully
2019-11-11T12:19:46.0225418Z ##[debug]defaultRoot: 'C:\vstsagent\A1\_work\_temp\temp_web_package_9699193652416063'
2019-11-11T12:19:46.0225668Z ##[debug]findOptions.allowBrokenSymbolicLinks: 'false'
2019-11-11T12:19:46.0225791Z ##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
2019-11-11T12:19:46.0225928Z ##[debug]findOptions.followSymbolicLinks: 'true'
2019-11-11T12:19:46.0226038Z ##[debug]matchOptions.debug: 'false'
2019-11-11T12:19:46.0226147Z ##[debug]matchOptions.nobrace: 'true'
2019-11-11T12:19:46.0226280Z ##[debug]matchOptions.noglobstar: 'false'
2019-11-11T12:19:46.0226387Z ##[debug]matchOptions.dot: 'true'
2019-11-11T12:19:46.0226547Z ##[debug]matchOptions.noext: 'false'
2019-11-11T12:19:46.0226657Z ##[debug]matchOptions.nocase: 'true'
2019-11-11T12:19:46.0226763Z ##[debug]matchOptions.nonull: 'false'
2019-11-11T12:19:46.0226896Z ##[debug]matchOptions.matchBase: 'false'
2019-11-11T12:19:46.0227016Z ##[debug]matchOptions.nocomment: 'false'
2019-11-11T12:19:46.0227146Z ##[debug]matchOptions.nonegate: 'false'
2019-11-11T12:19:46.0227252Z ##[debug]matchOptions.flipNegate: 'false'
2019-11-11T12:19:46.0227392Z ##[debug]pattern: '**\OURPROJECT\obj\Release\Package\PackageTmp\Web.config'
2019-11-11T12:19:46.0227519Z ##[debug]findPath: 'C:\vstsagent\A1\_work\_temp\temp_web_package_9699193652416063'
2019-11-11T12:19:46.0227632Z ##[debug]statOnly: 'false'
2019-11-11T12:19:46.0227769Z ##[debug]findPath: 'C:\vstsagent\A1\_work\_temp\temp_web_package_9699193652416063'
2019-11-11T12:19:46.0227885Z ##[debug]findOptions.allowBrokenSymbolicLinks: 'false'
2019-11-11T12:19:46.0228032Z ##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
2019-11-11T12:19:46.0228152Z ##[debug]findOptions.followSymbolicLinks: 'true'
...
2019-11-11T12:19:46.0970036Z ##[debug]615 results
2019-11-11T12:19:46.0970193Z ##[debug]found 615 paths
2019-11-11T12:19:46.0970374Z ##[debug]applying include pattern
2019-11-11T12:19:46.0970559Z ##[debug]adjustedPattern: 'C:\vstsagent\A1\_work\_temp\temp_web_package_9699193652416063\**\PackageTmp\Web.config'
2019-11-11T12:19:46.1099930Z ##[debug]1 matches
2019-11-11T12:19:46.1100207Z ##[debug]1 final results
2019-11-11T12:19:46.1172474Z ##[warning]Unable to apply transformation for the given package. Verify the following.
2019-11-11T12:19:46.1179673Z ##[debug]Processed: ##vso[task.issue type=warning;]Unable to apply transformation for the given package. Verify the following.
As the configuration in the config files you shared above, there's no any problem in it when apply it on my side.
In fact, in your build log, it has been show you the problem caused by what.
You can see that the task only found out only one file while here it should match 2 files, like this:
These 2 files, one is config and another is transformation config file. But in your build, it can only found out one file (Not sure whether the log you shared is completed, as I see, it only detect out Web.config file).
This issue should relevant the configuration of your task. Seems you were linking the artifact, and want to applying the transformation into that.
Please modify the Package or folder blank as this pic shown:
And change your Transformation rules as:
-transform **\*.Elastic.config -xml **\*.config
I'm trying to use MSDeploy with the parameter settings file option so I can build once and deploy to multiple environments by overriding the parameters with different files. From PowerShell I'm calling msdeploy.
msdeploy.exe -verb:sync `
-source:"contentPath='$SourceLocalPath'" `
-dest:"contentPath='$TargetLocalPath',computername='$TargetServer'"
-setParamFile:"$ParamFilePath" `
-verbose
This results in an error about not recognizing parameters.
msdeploy.exe : Error: The declared parameter 'SqlConnString' is not
recognized.
If I remove the "setParamFile" line, it deploys fine, but then uses default values. Also if I try to manually import the package from IIS, it displays the parameters with defaults filled in.
I have a Parameter.xml file in the root of the web project:
<parameters>
<parameter name="SqlConnString" description="Please provide the SQL connection string" defaultValue="...;Initial Catalog=xxx;server=xxx;" tags="">
<parameterEntry kind="XmlFile" scope="\\web.config$" match="/configuration/connectionStrings/add[#name='Sql']/#connectionString" />
</parameter>
</parameters>
The package is getting created with a ...SetParameters.xml file inside the package, which contains the entries from my Parameters.xml file plus the standard entries.
<parameters>
<setParameter name="SqlConnString" value="...Initial Catalog=xxx;server=xxx;"/>
...
</parameters>
Thank you
Try changing the value in "name" from "SqlConnString" to "YourSqlConnectionName-Web.config Connection String". If you want to know the exact name you have to look at your web.config file. The above example should work for a section like this in your web.config file:
<connectionStrings>
<add name="YourSqlConnectionName"
connectionString="...;Initial Catalog=xxx;server=xxx;"
providerName="System.Data.EntityClient"/>
</connectionStrings>
So you should configure your setParameters.xml file like:
<parameters>
<setParameter name="YourSqlConnectionName-Web.config Connection String" value="...Initial Catalog=xxx;server=xxx;"/>
</parameters>
and your Parameters.xml file like:
<parameters>
<parameter name="YourSqlConnectionName-Web.config Connection String" description="Please provide the SQL connection string" defaultValue="...;Initial Catalog=xxx;server=xxx;" tags="">
<parameterEntry kind="XmlFile" scope="\\web.config$" match="/configuration/connectionStrings/add[#name='YourSqlConnectionName']/#connectionString" />
</parameter>
</parameters>
Hope this solves your issue.