Is there a way to extract each matching result of a sed script to a file each? - text-processing

I have a file structured like the one below, only with more sections:
SCN DD1251 At Glasgow Road - Kilbowie Road
Modified By ________
Type CR
Region WS Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
DD1271 C DD1271 R
DD1351 D DD1351 B
E
SCN DD1271 At Glasgow Road - Hume Street
Modified 13-OCT-15 15:06 By BDAVIDSON
Type CR
Region WS Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
DD1301 T DD1301 A
DD1251 R DD1251 C
SCN DD1301 At Glasgow Road - Argyll Road - Cart Street
Modified 27-JUN-16 11:45 By BDAVIDSON
Type CR
Region WS Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
DD1271 A DD1271 T
B
C
D
SCN DD1351 At Kilbowie Road - Chalmers Street
Modified 9-FEB-21 08:57 By BDAVIDSON
Type CR
Region WS Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
A DD1251 D
DD1251 B
C
SCN DD1451 At Kilbowie Road - Montrose Street
Modified 9-FEB-21 08:58 By BDAVIDSON
Type CR
Region WC Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
A
B
C
SCN DD1601 At Kilbowie Road - Thomson Street 3G
Modified 9-FEB-21 08:59 By BDAVIDSON
Type CR
Region WV Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
A
B
C
D
SCN DD1621 At Argyll Road - Stanford Street 3G
Modified 9-FEB-21 08:59 By BDAVIDSON
Type CR
Region AS Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
B
C
D
Currently, I am using the following bash script containing a sed function to pull out each section of data from one SCN to the line of no characters between entires:
sed -n '/^SCN/,/^\s*$/p' LOG.TXT > Junction-Links.txt
This produces the following output:
SCN DD1251 At Glasgow Road - Kilbowie Road
Modified By ________
Type CR
Region WS Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
DD1271 C DD1271 R
DD1351 D DD1351 B
E
SCN DD1271 At Glasgow Road - Hume Street
Modified 13-OCT-15 15:06 By BDAVIDSON
Type CR
Region WS Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
DD1301 T DD1301 A
DD1251 R DD1251 C
My question is, is it possible to save each matching pattern to a separate file (i.e a file for SCN DD1251, a file for SCN DD1271 etc.) rather than saving them all into one combined file?
Ideally the output would be like so:
SCN DD1251.txt
SCN DD1271.txt
SCN DD1301.txt
SCN DD1351.txt
With SCN DD1251.txt simply containing the following:
SCN DD1251 At Glasgow Road - Kilbowie Road
Modified By ________
Type CR
Region WS Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
DD1271 C DD1271 R
DD1351 D DD1351 B
E

Using any awk in any shell on every Unix box:
$ awk '/^SCN/{close(out); out=$1" "$2".txt"} {print > out}' file
$ head SCN*
==> SCN DD1251.txt <==
SCN DD1251 At Glasgow Road - Kilbowie Road
Modified By ________
Type CR
Region WS Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
DD1271 C DD1271 R
DD1351 D DD1351 B
E
==> SCN DD1271.txt <==
SCN DD1271 At Glasgow Road - Hume Street
Modified 13-OCT-15 15:06 By BDAVIDSON
Type CR
Region WS Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
DD1301 T DD1301 A
DD1251 R DD1251 C
==> SCN DD1301.txt <==
SCN DD1301 At Glasgow Road - Argyll Road - Cart Street
Modified 27-JUN-16 11:45 By BDAVIDSON
Type CR
Region WS Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
DD1271 A DD1271 T
B
C
D
==> SCN DD1351.txt <==
SCN DD1351 At Kilbowie Road - Chalmers Street
Modified 9-FEB-21 08:57 By BDAVIDSON
Type CR
Region WS Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
A DD1251 D
DD1251 B
C
==> SCN DD1451.txt <==
SCN DD1451 At Kilbowie Road - Montrose Street
Modified 9-FEB-21 08:58 By BDAVIDSON
Type CR
Region WC Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
A
B
C
==> SCN DD1601.txt <==
SCN DD1601 At Kilbowie Road - Thomson Street 3G
Modified 9-FEB-21 08:59 By BDAVIDSON
Type CR
Region WV Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
A
B
C
D
==> SCN DD1621.txt <==
SCN DD1621 At Argyll Road - Stanford Street 3G
Modified 9-FEB-21 08:59 By BDAVIDSON
Type CR
Region AS Subregion
UPSTREAM DOWNSTREAM FILTER
NODE LINK NODE LINK LINK
B
C
D

You could use csplit:
csplit -z -f 'SCN DD' Junction-Links.txt '/^\s*$/' '{*}'
Although this will make files named 'SCN DD00', 'SCN DD01', 'SCN DD02' etc. If
you have more than 100 files, you can add -n 3 to use a three digit padding
in the output files.

Related

Showing Agent Names side by side in tableau

I have 2 columns first one Id and another is Agent. Each Id has different Agents.
Now I need to show names besides each ID.
But I am getting data like below.
I kept ID and name in rows shelf
Expected out put :
ID Name Name
12345 Ram Rajesh
4444 Ravi Raghu
3333 Reena Teena James
Out put I am getting :
ID Name
12345 Ram
12345 Rajesh
4444 Ravi
4444 Raghu
3333 Reena
3333 Teena
3333 James

Google App Engine respond with the code 308

Problem
1.I have a sns service which is running on Google App Engine
2.The sns service is using go to run as the back ending side
3.The sns service is using a crob job to request Api A
4.The Api A is adding another http request(request to Api B) into taskqueue
5.The taskqueue is executed immediately after the request of Api A
6.The request of Api B is executed
7.The Api B responded with the code 308
Problem Summary
job cron(request for Api A) -> Api A -> Task queue(stack request for Api B) ->Api B -> some execution
Api A responded with 200
Api B responded with 308
Question
Why Api B is respond with the code of 308?
Complement
By the way the following sequence is successful
post man (request for Api A) -> Api A -> Task queue(request for Api B) ->Api B -> some execution
Api A responded with 200
Api B responded with 200
When I use post man to request Api A the following request for Api B was respond with 200
The way I did with taskqueue is like
params := url.Values{}
params.Set("dollSpaces", strings.Join(dollSpaces, ","))
t := taskqueue.NewPOSTTask("/path", params)
req.Header.Set("X-AppEngine-QueueName", "mailqueue")
c := appengine.NewContext(req)
taskqueue.Add(c, t, "my queue")
The Log from Google Cloud Platform is like the following
Response of Api A
2019-01-01 10:00:00.001 JST
GET
200
193 B
577 ms
AppEngine-Google; (+http://code.google.com/appengine)
/path/ApiAPath
0.1.0.1 - - [01/Jan/2019:10:00:00 +0900] "GET /path/ApiAPath HTTP/1.1" 200 193 - "AppEngine-Google; (+http://code.google.com/appengine)" "aaa.aaa.com" ms=577 cpu_ms=3 cpm_usd=2.8039999999999998e-8 loading_request=0 instance=asda1231313asdasd123123asda123 app_engine_release=1.9.71 trace_id=123123123dsfasf123123zafasdasd
Response of Api B
2019-01-01 10:00:00.100 JST
POST
308
95 B
2 ms
AppEngine-Google; (+http://code.google.com/appengine)
/ApiBPath
0.1.0.2 - - [01/Jan/2019:10:00:00 +0900] "POST /ApiBPath HTTP/1.1" 308 95 http://aaa.aaa.com/path/ApiAPath "AppEngine-Google; (+http://code.google.com/appengine)" "aaa.aaa.com" ms=2 cpu_ms=221 cpm_usd=1.3802e-8 loading_request=0 instance= asda1231313asdasd123123asda123 app_engine_release=1.9.71 trace_id= 123123123dsfasf123123zafasdasd

Is there a legitimate reason Amazon probes /admin

From my raw server logs, I found these:
52.42.136.103 02/Aug/2016:17:58:12 GET /administrator/
54.234.186.225 03/Aug/2016:16:48:55 GET /administrator/
54.149.57.2 04/Aug/2016:18:40:55 GET /administrator/
52.40.119.32 06/Aug/2016:09:34:37 GET /administrator/
52.40.119.32 08/Aug/2016:06:02:45 GET /administrator/
The ip's all belong to Amazon Technologies (and I don't have an /admin directory).
I'm used to seeing that kind of random probe thing (/wp-admin, /wp-blog/, /dir, ... ) from Russia and Vietnam and so on, and I've always assumed there was some nefarious motive behind it. Is there? Why would Amazon be doing that? Is Jeff Bezos out of control?
Amazon itself is not crawling your site but more likely to be some of their EC2 users (i.e. rented servers on the cloud). See below for email address to contact Amazon if you think their users are doing something illegal or abusive
whois 52.40.119.32
#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/whois_tou.html
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/public/whoisinaccuracy/index.xhtml
#
#
# The following results may also be obtained via:
# https://whois.arin.net/rest/nets;q=52.40.119.32?showDetails=true&showARIN=false&showNonArinTopLevelNet=false&ext=netref2
#
NetRange: 52.32.0.0 - 52.63.255.255
CIDR: 52.32.0.0/11
NetName: AT-88-Z
NetHandle: NET-52-32-0-0-1
Parent: NET52 (NET-52-0-0-0-0)
NetType: Direct Allocation
OriginAS:
Organization: Amazon Technologies Inc. (AT-88-Z)
RegDate: 2015-09-02
Updated: 2015-09-02
Ref: https://whois.arin.net/rest/net/NET-52-32-0-0-1
OrgName: Amazon Technologies Inc.
OrgId: AT-88-Z
Address: 410 Terry Ave N.
City: Seattle
StateProv: WA
PostalCode: 98109
Country: US
RegDate: 2011-12-08
Updated: 2014-10-20
Comment: All abuse reports MUST include:
Comment: * src IP
Comment: * dest IP (your IP)
Comment: * dest port
Comment: * Accurate date/timestamp and timezone of activity
Comment: * Intensity/frequency (short log extracts)
Comment: * Your contact details (phone and email) Without these we will be unable to identify the correct owner of the IP address at that point in time.
Ref: https://whois.arin.net/rest/org/AT-88-Z
OrgTechHandle: ANO24-ARIN
OrgTechName: Amazon EC2 Network Operations
OrgTechPhone: +1-206-266-4064
OrgTechEmail: amzn-noc-contact#amazon.com
OrgTechRef: https://whois.arin.net/rest/poc/ANO24-ARIN
OrgAbuseHandle: AEA8-ARIN
OrgAbuseName: Amazon EC2 Abuse
OrgAbusePhone: +1-206-266-4064
OrgAbuseEmail: abuse#amazonaws.com
OrgAbuseRef: https://whois.arin.net/rest/poc/AEA8-ARIN
OrgNOCHandle: AANO1-ARIN
OrgNOCName: Amazon AWS Network Operations
OrgNOCPhone: +1-206-266-4064
OrgNOCEmail: amzn-noc-contact#amazon.com
OrgNOCRef: https://whois.arin.net/rest/poc/AANO1-ARIN

AODV Routing Protocol - Route Reply

I have a AODV pcap file which shows me some RREP packets with both destination and Originator IP address to be the same.
Is it correct or this could possibly be an error.
My sample pcap file can be found here
RREP with same D and O are hello messages

How can I implement a simple WHOIS proxy in Perl?

I have several WHOIS servers for which I want to have a single proxy. The proxy should forward requests to appropriate servers based on the data in the query. How to approach this problem?
Step 1 read the RFC for WHOIS
Step 2 implement a mock server with Net::Server
Step 3 implement the proxy using your mock server and Net::Whois::Proxy or some other WHOIS module
A quick scan of the RFC seems to say that it is just a simple text based protocol that expects one CRLF terminated line and then sends a blob of CRLF terminate lines and closes the socket.
Yep, it really looks that simple, given this code
#!/usr/bin/perl
{
package Whois;
use strict;
use warnings;
use parent 'Net::Server';
sub process_request {
my $request = <>;
print "you sent me $request";
}
}
Whois->run;
you can say
whois -h localhost -p 20203 foo.com
and get back
you sent me foo.com
Given the "high-load" tag, you will probably want to switch over to Net::Server::PreForkSimple personality when you are done with testing.
And just because I was bored:
#!/usr/bin/perl
package Whois {
use strict;
use warnings;
use parent 'Net::Server::PreFork';
use Net::Whois::Raw;
my %handler = (
org => "whois.publicinterestregistry.net",
);
sub process_request {
(my $request = <>) =~ s/[.]([^.]+)\x{0d}\x{0a}/.$1/;
print exists $handler{$1} ?
whois $request, $handler{$1} :
"I don't know where to look for $request\r\n";
}
}
Whois->run(
user => "nobody",
group => "nobody",
port => 43,
min_servers => 1, #min number of children
max_servers => 10, #max number of children
min_spare_servers => 1, #fork if we don't have this many waiting
max_spare_servers => 5, #kill if we have this many waiting
max_requests => 10_000, #num of requests before killing a child
);
which when I run
sudo perl whois.pl
and then
whois -h localhost foo.org
gives us
NOTICE: Access to .ORG WHOIS information is provided to assist persons in
determining the contents of a domain name registration record in the Public Interest Registry
registry database. The data in this record is provided by Public Interest Registry
for informational purposes only, and Public Interest Registry does not guarantee its
accuracy. This service is intended only for query-based access. You agree
that you will use this data only for lawful purposes and that, under no
circumstances will you use this data to: (a) allow, enable, or otherwise
support the transmission by e-mail, telephone, or facsimile of mass
unsolicited, commercial advertising or solicitations to entities other than
the data recipient's own existing customers; or (b) enable high volume,
automated, electronic processes that send queries or data to the systems of
Registry Operator or any ICANN-Accredited Registrar, except as reasonably
necessary to register domain names or modify existing registrations. All
rights reserved. Public Interest Registry reserves the right to modify these terms at any
time. By submitting this query, you agree to abide by this policy.
Domain ID:D1608104-LROR
Domain Name:FOO.ORG
Created On:10-Jan-1995 05:00:00 UTC
Last Updated On:07-Mar-2011 00:26:43 UTC
Expiration Date:09-Jan-2012 05:00:00 UTC
Sponsoring Registrar:Fabulous.com Pty Ltd. (R133-LROR)
Status:CLIENT DELETE PROHIBITED
Status:CLIENT TRANSFER PROHIBITED
Registrant ID:fabwpp-000700385
Registrant Name:Domain Hostmaster, CustomerID : 85519846801225
Registrant Organization:Whois Privacy Services Pty Ltd
Registrant Street1:PO Box 923
Registrant Street2:
Registrant Street3:
Registrant City:Fortitude Valley
Registrant State/Province:QLD
Registrant Postal Code:4006
Registrant Country:AU
Registrant Phone:+61.730070090
Registrant Phone Ext.:
Registrant FAX:+61.730070091
Registrant FAX Ext.:
Registrant Email:85519846801225-959797#whoisprivacyservices.com.au
Admin ID:fabwpp-000700385
Admin Name:Domain Hostmaster, CustomerID : 85519846801225
Admin Organization:Whois Privacy Services Pty Ltd
Admin Street1:PO Box 923
Admin Street2:
Admin Street3:
Admin City:Fortitude Valley
Admin State/Province:QLD
Admin Postal Code:4006
Admin Country:AU
Admin Phone:+61.730070090
Admin Phone Ext.:
Admin FAX:+61.730070091
Admin FAX Ext.:
Admin Email:85519846801225-959797#whoisprivacyservices.com.au
Tech ID:fabwpp-000700385
Tech Name:Domain Hostmaster, CustomerID : 85519846801225
Tech Organization:Whois Privacy Services Pty Ltd
Tech Street1:PO Box 923
Tech Street2:
Tech Street3:
Tech City:Fortitude Valley
Tech State/Province:QLD
Tech Postal Code:4006
Tech Country:AU
Tech Phone:+61.730070090
Tech Phone Ext.:
Tech FAX:+61.730070091
Tech FAX Ext.:
Tech Email:85519846801225-959797#whoisprivacyservices.com.au
Name Server:NS1.HITFARM.COM
Name Server:NS2.HITFARM.COM
Name Server:
Name Server:
Name Server:
Name Server:
Name Server:
Name Server:
Name Server:
Name Server:
Name Server:
Name Server:
Name Server:
DNSSEC:Unsigned
whois.publicinterestregistry.net
I'm no expert on whois, but Net::Whois::Proxy would seem to be what you want.