Merge lines between two patterns using sed - sed

I have an output file that looks like this:
HEADER 1:
server1 server2 server3
server4 server5 server6
server7 server8 server9
HEADER 2:
HEADER 1:
server10 server11 server12
server13 server14 server15
server16 server17 server18
HEADER 2:
I need to merge everything between the two headers so the updated output file looks like this:
HEADER 1:
server1 server2 server3 server4 server5 server6 server7 server8 server9
HEADER 2:
HEADER 1:
server10 server11 server12 server13 server14 server15 server16 server17 server18
HEADER 2:
The header names always remain constant.
I'm using UnxUtils in Windows, so unfortunately I don't have awk to do the manipulation. How do I go about it using sed?

Using sed:
sed '/HEADER 1/{n;:l N;/HEADER 2/b; s/\n//; bl}' input
n skips/prints the current line (HEADER 1), clears buffer
l is a label for looping (goto label)
N adds (appends) lines to buffer (preserving newlines)
/HEADER 2/b, where b is branch (without the label, it is break), break out when HEADER2 is seen
s/\n// removes the newlines in the buffer
bl jumps back to label l

This shows how it could be done using awk
You can download Gawk for Windows
awk '/HEADER/ {printf "\n%s\n",$0;next} {printf "%s ",$0}' file
HEADER 1:
server1 server2 server3 server4 server5 server6 server7 server8 server9
HEADER 2:
HEADER 1:
server10 server11 server12 server13 server14 server15 server16 server17 server18
HEADER 2:

Related

CentOS EPEL fail2ban not processing systemd journal for tomcat

I've installed fail2ban 0.10.5-2.el7 from EPEL on CentOS 7.8. I'm trying to get it to work with systemd for processing a Tomcat log (also systemd).
In jail.local I added:
[guacamole]
enabled = true
port = http,https
backend = systemd
In filter.d/guacamole.conf:
[Definition]
failregex = Authentication attempt from <HOST> for user "[^"]*" failed\.$
ignoreregex =
journalmatch = _SYSTEMD_UNIT=tomcat.service + _COMM=java
If I run journalctl -u tomcat.service I see all the log lines. The ones I am interested in look like this:
May 18 13:58:26 myhost catalina.sh[42065]: 13:58:26.485 [http-nio-8080-exec-6] WARN o.a.g.r.auth.AuthenticationService - Authentication attempt from 1.2.3.4 for user "test" failed.
If I redirect journalctl -u tomcat.service to a log file, and process it with fail2ban-regex then it works exactly the way I want it to work, finding all the lines it needs.
% fail2ban-regex /tmp/j9 /etc/fail2ban/filter.d/guacamole.conf
Running tests
=============
Use failregex filter file : guacamole, basedir: /etc/fail2ban
Use log file : /tmp/j9
Use encoding : UTF-8
Results
=======
Failregex: 47 total
|- #) [# of hits] regular expression
| 1) [47] Authentication attempt from <HOST> for user "[^"]*" failed\.$
`-
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [1] ExYear(?P<_sep>[-/.])Month(?P=_sep)Day(?:T| ?)24hour:Minute:Second(?:[.,]Microseconds)?(?:\s*Zone offset)?
| [570] {^LN-BEG}(?:DAY )?MON Day %k:Minute:Second(?:\.Microseconds)?(?: ExYear)?
`-
Lines: 571 lines, 0 ignored, 47 matched, 524 missed
[processed in 0.12 sec]
However, if fail2ban reads the journal directly then it does not work:
fail2ban-regex systemd-journal /etc/fail2ban/filter.d/guacamole.conf
It comes back right away, and processes 0 lines!
Running tests
=============
Use failregex filter file : guacamole, basedir: /etc/fail2ban
Use systemd journal
Use encoding : UTF-8
Use journal match : _SYSTEMD_UNIT=tomcat.service + _COMM=java
Results
=======
Failregex: 0 total
Ignoreregex: 0 total
Lines: 0 lines, 0 ignored, 0 matched, 0 missed
[processed in 0.00 sec]
I've tried to remove _COMM=java. It doesn't make a difference.
If I leave out the journal match line altogether, it at least processes all the lines from the journal, but does not find any matches (even though, as I mentioned, it processes a dump of the log file fine):
Running tests
=============
Use failregex filter file : guacamole, basedir: /etc/fail2ban
Use systemd journal
Use encoding : UTF-8
Results
=======
Failregex: 0 total
Ignoreregex: 0 total
Lines: 202271 lines, 0 ignored, 0 matched, 202271 missed
[processed in 34.54 sec]
Missed line(s): too many to print. Use --print-all-missed to print all 202271 lines
Either this is a bug, or I'm missing a small detail.
Thanks for any help you can provide.
To make sure the filter definition is properly initialised, it would be good to include the common definition. Your filter definition (/etc/fail2ban/filter.d/guacamole.conf) would therefore look like:
[INCLUDES]
before = common.conf
[Definition]
journalmatch = _SYSTEMD_UNIT='tomcat.service'
failregex = Authentication attempt from <HOST> for user "[^"]*" failed\.$
ignoreregex =
A small note given that your issue only occurs with systemd but not flat files, could you try the same pattern without $ at the end? Maybe there is an issue with the end of line when printed to the journal?
In your jail definition (/etc/fail2ban/jail.d/guacamole.conf), remember to define the ban time/find time/retries if they haven't already been defined in the default configuration:
[guacamole]
enabled = true
port = http,https
maxretry = 3
findtime = 1h
bantime = 1d
# "backend" specifies the backend used to get files modification.
# systemd: uses systemd python library to access the systemd journal.
# Specifying "logpath" is not valid for this backend.
# See "journalmatch" in the jails associated filter config
backend = systemd
Remember to restart the fail2ban service after doing such changes.

Sh - how to remove all after selection

I'm trying to manipulate a string and the first step is to truncate the variable after a combination, so first I create the var:
SERVER_CONFIGURATION_FILE=$(curl some remove url)
And now my $SERVER_CONFIGURATION_FILE is something like that
client
dev tun
proto udp
remote 192.145.127.237 1194
resolv-retry infinite
remote-random
nobind
tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
persist-key
persist-tun
ping 15
ping-restart 0
ping-timer-rem
reneg-sec 0
comp-lzo no
remote-cert-tls server
auth-user-pass
verb 3
pull
fast-io
cipher AES-256-CBC
auth SHA512
<ca>
-----BEGIN CERTIFICATE-----
MIIFCjCCAvKgAwIBAgIBATANBgkqhkiG9w0BAQ0FADA5MQswCQYDVQQGEwJQQTEQ
MA4GA1UEChMHTm9yZFZQTjEYMBYGA1UEAxMPTm9yZFZQTiBSb290IENBMB4XDTE2
MDEwMTAwMDAwMFoXDTM1MTIzMTIzNTk1OVowOTELMAkGA1UEBhMCUEExEDAOBgNV
BAoTB05vcmRWUE4xGDAWBgNVBAMTD05vcmRWUE4gUm9vdCBDQTCCAiIwDQYJKoZI
hvcNAQEBBQADggIPADCCAgoCggIBAMkr/BYhyo0F2upsIMXwC6QvkZps3NN2/eQF
kfQIS1gql0aejsKsEnmY0Kaon8uZCTXPsRH1gQNgg5D2gixdd1mJUvV3dE3y9FJr
XMoDkXdCGBodvKJyU6lcfEVF6/UxHcbBguZK9UtRHS9eJYm3rpL/5huQMCppX7kU
eQ8dpCwd3iKITqwd1ZudDqsWaU0vqzC2H55IyaZ/5/TnCk31Q1UP6BksbbuRcwOV
skEDsm6YoWDnn/IIzGOYnFJRzQH5jTz3j1QBvRIuQuBuvUkfhx1FEwhwZigrcxXu
MP+QgM54kezgziJUaZcOM2zF3lvrwMvXDMfNeIoJABv9ljw969xQ8czQCU5lMVmA
37ltv5Ec9U5hZuwk/9QO1Z+d/r6Jx0mlurS8gnCAKJgwa3kyZw6e4FZ8mYL4vpRR
hPdvRTWCMJkeB4yBHyhxUmTRgJHm6YR3D6hcFAc9cQcTEl/I60tMdz33G6m0O42s
Qt/+AR3YCY/RusWVBJB/qNS94EtNtj8iaebCQW1jHAhvGmFILVR9lzD0EzWKHkvy
WEjmUVRgCDd6Ne3eFRNS73gdv/C3l5boYySeu4exkEYVxVRn8DhCxs0MnkMHWFK6
MyzXCCn+JnWFDYPfDKHvpff/kLDobtPBf+Lbch5wQy9quY27xaj0XwLyjOltpiST
LWae/Q4vAgMBAAGjHTAbMAwGA1UdEwQFMAMBAf8wCwYDVR0PBAQDAgEGMA0GCSqG
SIb3DQEBDQUAA4ICAQC9fUL2sZPxIN2mD32VeNySTgZlCEdVmlq471o/bDMP4B8g
nQesFRtXY2ZCjs50Jm73B2LViL9qlREmI6vE5IC8IsRBJSV4ce1WYxyXro5rmVg/
k6a10rlsbK/eg//GHoJxDdXDOokLUSnxt7gk3QKpX6eCdh67p0PuWm/7WUJQxH2S
DxsT9vB/iZriTIEe/ILoOQF0Aqp7AgNCcLcLAmbxXQkXYCCSB35Vp06u+eTWjG0/
pyS5V14stGtw+fA0DJp5ZJV4eqJ5LqxMlYvEZ/qKTEdoCeaXv2QEmN6dVqjDoTAo
k0t5u4YRXzEVCfXAC3ocplNdtCA72wjFJcSbfif4BSC8bDACTXtnPC7nD0VndZLp
+RiNLeiENhk0oTC+UVdSc+n2nJOzkCK0vYu0Ads4JGIB7g8IB3z2t9ICmsWrgnhd
NdcOe15BincrGA8avQ1cWXsfIKEjbrnEuEk9b5jel6NfHtPKoHc9mDpRdNPISeVa
wDBM1mJChneHt59Nh8Gah74+TM1jBsw4fhJPvoc7Atcg740JErb904mZfkIEmojC
VPhBHVQ9LHBAdM8qFI2kRK0IynOmAZhexlP/aT/kpEsEPyaZQlnBn3An1CRz8h0S
PApL8PytggYKeQmRhl499+6jLxcZ2IegLfqq41dzIjwHwTMplg+1pKIOVojpWA==
-----END CERTIFICATE-----
</ca>
key-direction 1
<tls-auth>
#
# 2048 bit OpenVPN static key
#
-----BEGIN OpenVPN Static key V1-----
e685bdaf659a25a200e2b9e39e51ff03
0fc72cf1ce07232bd8b2be5e6c670143
f51e937e670eee09d4f2ea5a6e4e6996
5db852c275351b86fc4ca892d78ae002
d6f70d029bd79c4d1c26cf14e9588033
cf639f8a74809f29f72b9d58f9b8f5fe
fc7938eade40e9fed6cb92184abb2cc1
0eb1a296df243b251df0643d53724cdb
5a92a1d6cb817804c4a9319b57d53be5
80815bcfcb2df55018cc83fc43bc7ff8
2d51f9b88364776ee9d12fc85cc7ea5b
9741c4f598c485316db066d52db4540e
212e1518a9bd4828219e24b20d88f598
a196c9de96012090e333519ae18d3509
9427e7b372d348d352dc4c85e18cd4b9
3f8a56ddb2e64eb67adfc9b337157ff4
-----END OpenVPN Static key V1-----
</tls-auth>
From here I want to delete all after <ca> tag
I tried various way, but every time the result is the same, I don't know what I do wrong
The last I tried is
echo "$SERVER_CONFIGURATION_FILE" | sed 's/<c[.\s\S\d\D\w\W]*//'
but remove only <ca and nothing more and the funny is that under regex101 this regex works
I feel pretty stupid but I can't find what I miss!
Any idea?
Please note that I'm under sh with busybox 1.30
Is this what you're trying to do?
$ awk '/<ca>/{exit} 1' file
client
dev tun
proto udp
remote 192.145.127.237 1194
resolv-retry infinite
remote-random
nobind
tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
persist-key
persist-tun
ping 15
ping-restart 0
ping-timer-rem
reneg-sec 0
comp-lzo no
remote-cert-tls server
auth-user-pass
verb 3
pull
fast-io
cipher AES-256-CBC
auth SHA512
if your text is in a variable or coming from a pipe otherwise then just call it like this:
echo "$SERVER_CONFIGURATION_FILE" | awk '/<ca>/{exit} 1'
You don't need sed or awk for that. You can do it directly in the shell like this:
SERVER_CONFIGURATION_FILE="${SERVER_CONFIGURATION_FILE%%<ca>*}"
(Tested in dash; I assume busybox sh works the same way.)
${var%%pattern} expands to the contents of var, but with the longest suffix matching the glob pattern pattern stripped away.
The main problem with your sed attempt is that sed reads input one line at a time, so matches from one line wouldn't affect the other lines. (Also, your regex is slightly crazy and doesn't work like that in sed anyway.)
A working sed solution:
sed '/<ca>/{x;q}'
Or, perhaps more directly:
sed -n '/<ca>/q; p'
I.e. do not print by default; if line matches <ca>, quit; otherwise print and read the next line.

Get console messages into a message box

Is there any way to get the Pure Data Pd windows console messages into a message box within the patch? I'm trying to get error messages back into the patch so that the patch can react to them.
the only way to do this (in Pd-vanilla even) is to:
send all post/error messages to the stderr (with the -stderr startup flag)
redirect the stderr to stdout
capture the stdout with a script, reformat it and send it back to Pd via the network (preferrably UDP)
add a [netreceive] to get the messages.
something like the following:
$ pd -stderr 2>&1 | while true; do pdsend 9999 localhost udp; done
and in Pd add the following to your patch:
[netreceive 9999 1]
|
[route error:]
|
[list prepend set]
|
[list trim]
|
[ (
now whenever Pd prints a line starting with "error:" this line will appear in the msgbox.
caveats: this effectively suppresses all messages on the Pd-console.

python formatting return value of subprocess

I am attempting on Python 2.6.6 to get the routing table of a system (for a interface) into a python list to parse; but I cannot seem to solve why the entire result is stored into one variable.
The loop seems to iterate over one characters at a time, while the behavior I wanted was one line at a time.
what I get is one character; short example below...
1
0
.
2
4
3
what I'd like line to return; so I can run other commands against each line..
10.243.186.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
10.243.188.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
10.243.184.0 10.243.186.1 255.255.255.128 UG 0 0 0 eth0
Here is the code below...
def getnet(int):
int = 'eth0' ####### for testing only
cmd = ('route -n | grep ' + int)
routes = subprocess.Popen([cmd], shell=True, stdout=subprocess.PIPE)
routes, err = routes.communicate()
for line in routes:
print line
routes in your case is a bytestring that contains the entire output from the shell command. for character in astring statement produces one character at a time as the example in your question demonstrates. To get lines as a list of strings instead, call lines = all_output.splitlines():
from subprocess import check_output
lines = check_output("a | b", shell=True, universal_newlines=True).splitlines()
Here's a workaround if your Python version has no check_output(). If you want to read one line at a time while the process is still running, see Python: read streaming input from subprocess.communicate().
You could try to use os, ctypes modules, to get the info instead of grepping the output of an external command.

print first column and print everything after % sign

I want to print the first column in this case the host IP and everything after % character.
10.162.254.12 3558: 003557: Mar 4 2013 12:13:28: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface
GigabitEthernet8/37, changed state to up
192.50.0.14 : 2013 Mar 4 13:14:20 EST: %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet17/33 is down (Link
failure)
You have two % characters and didn't specify if everything after the first or last should be printed? Here is both ways using sed:
# First field and everything after first %
$ sed 's/ [^%]*%/ /' file
10.162.254.12 LINEPROTO-SP-5-UPDOWN: Line protocol on Interface GigabitEthernet8/37, changed state to up 192.50.0.14 : 2013 Mar 4 13:14:20 EST: %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet17/33 is down (Link failure)
# First field and everything after last %
$ sed 's/ .*%/ /' file
10.162.254.12 ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet17/33 is down (Link failure)
sed 's/\([^ ]*\)[^%]*%/\1 /' input