Search after match in dynamic lines [closed] - perl

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 4 years ago.
Improve this question
Dears,
I have the file below in the following format
Success|Filter passed|[invalid field]|[invalid field]|Id-350a875b087965e58cbe1f4a
Accept: text/plain, text/plain, application/json, application/*+json, */*, */*
Host: api2.tim.com.br
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Via: 1.1
X-Forwarded-For: 144.22.98.123
X-Forwarded-Host:
X-Forwarded-Server:
Success|Success in calling policy shortcut|[invalid field]|[invalid field]|[invalid field]|Id-350a875b087965e58cbe1f4a|Call 'Set Request Message'|GET
Accept: text/plain, text/plain, application/json, application/*+json, */*, */*
Host: api2.tim.com.br
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Via: 1.1 api2.tim.com.br
X-Forwarded-For: 144.22.98.123
X-Forwarded-Host:
X-Forwarded-Server:
Content-Type: text/xml; charset="UTF-8"
I need to perform a search for the line that begins with the string "^ Success" and display all the items until there is a "^ Sucess" string again.
Here is an example of what I need to display:
Success|Filter passed|[invalid field]|[invalid field]|Id-350a875b087965e58cbe1f4a
Accept: text/plain, text/plain, application/json, application/*+json, */*, */*
Host: api2.tim.com.br
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Via: 1.1
X-Forwarded-For: 144.22.98.123
X-Forwarded-Host:
X-Forwarded-Server:
What happens is that the amount of lines below after the match is very dynamic.
What happens is that the amount of lines below the match is very dynamic and in the same file there may be several lines
of the same match and I would need to display them also when the file is run.
Could you guys help me?

Perl has a "paragraph mode". You change the input record separator, $/ to read chunks of "multiline" text. This splits up your data on the double newline:
use v5.10;
$/ = "\n\n";
while( <INPUT> ) {
chomp;
say "==========\n$_\n----------\n";
}
Start your program with that and try to do whatever else you're trying to do. In your next question you'll have the small demonstration program you need to get better help.

Related

Setup Fail2ban for a specifc url

For shits and giggles I created a small honepot php script. If it is called from a webpage, I want to simply put the IP address in jail.
I created a filter that looks like this
filename: apache-specific-url.conf
[INCLUDES]
before = apache-common.conf
[Definition]
failregex = ^<HOST> -.*"(GET|POST).*\/sshlogin.php\/.*$
ignoreregex =
I've also put the following into my jail.local
[apache-specific-url]
enabled = true
port = http,https
filter = apache-specific-url
logpath = %(apache_access_log)s
bantime = 48h
maxretry = 1
Fail2ban shows that my jail is running. However, if I access it via domain.com/sshlogin.php or IPaddress/sshlogin.php... the URL never gets banned.
Is my regex the problem?
Is the filter the problem?
Is it that my mother didn't love me as a child?
Any help appreciated.
Tail of the log
111.111.111.111 - - [13/Jan/2021:15:05:16 -0500] "GET /sshlogin.php HTTP/1.1" 200 3548 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15"
111.111.111.111 - - [13/Jan/2021:15:05:19 -0500] "GET /sshlogin.php HTTP/1.1" 200 3548 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15"
111.111.111.111 - - [13/Jan/2021:15:05:20 -0500] "GET /sshlogin.php HTTP/1.1" 200 3548 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15"
111.111.111.111 - - [13/Jan/2021:15:05:25 -0500] "GET /sshlogin.php HTTP/1.1" 200 3548 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15"
The regex in your comment above definitely won't have any hits because it misses the most important part <HOST> and also specifies the end of the line immediately after the sshlogin.php part. The regex in your post is wrong only because you've included a redundant slash after the sshlogin.php part, otherwise it would match. However you'd also need to set a custom date pattern for that specific log, so use the following:
[INCLUDES]
before = apache-common.conf
[Definition]
failregex = ^<HOST> - - \[[^\]]*\] "(GET|POST) /sshlogin\.php
ignoreregex =
datepattern = %%d/%%b/%%Y:%%H:%%M:%%S \-%%f
I changed the failregex to make it more specific and avoid unnecessary quantifiers which might get you in trouble.
Be sure to restart fail2ban after the changes.
(I'm sure your mother loved you as a child btw.)

how to download large file using Curlwget extension

I have added the CurlWget extension for my browser and tried to download data using jupyter notebook as below:
!wget --header="Host: storage.googleapis.com" --header="User-Agent: Mozilla/5.0 (Windows NT 10.0;
Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36
OPR/66.0.3515.44" --header="Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,
application/signed-exchange;v=b3;q=0.9" --header="Accept-Language: en-US,en;q=0.9" --
header="Referer: https://www.kaggle.com/" "https://storage.googleapis.com/kagglesdsdata/competitions
/4117/46665/train.7z?GoogleAccessId=web-data#kaggle-161607.iam.gserviceaccount.com&Expires=1580049706&Signature=kowWRCMZZkqsrEqcwFtNJd4nwGgpE9DLbAcJ2b%2BvaGw1Wie82k3K03bhmHpqnhIKPsloHQJRq%2FHpBxv4kSeINAymvKvJXcpffjMqx%2Baujazoqxbl0aAQUhBs27OTKTqSp5Hzfhpz%2FKd%2Fx6SuYUCxy7x%2BAFOjlzQ8se59vJPwEmRNr4%2BSeOepC%2F%2BWJYzgLIcXDFy%2BUjjH1SrnBdAgRiMEa8pPD%2FZxmRma4ggWIWskLEVyuq4oAyVnaXK%2F39GsCo5lr199KqsPsO7BYJxs2hGv%2FlY6n4PirdQpw68dsSrLvfnSbpQckVVRtqjb9uLWDsQqarWfec1INAmHwaa%2B2Db2yQ%3D%3D&response-content-disposition=attachment%3B+filename%3Dtrain.7z" -O "train.7z" -c
But i am getting below error:
'wget' is not recognized as an internal or external command, operable program or batch file.
i have installed wget using below command:
pip install wget
This probably not a satisfying answer, but the answer is don't do this. CurlWget was flagged as malware and taken from the chrome web store:
https://chrome.google.com/webstore/detail/curlwget/jmocjfidanebdlinpbcdkcmgdifblncg/support?hl=pt-BR&authuser=2

Where can I set the useragent in Strawberry Perl's config?

We have a proxy server here and all internet traffic is going through that. The command: cpan package fails with the following error:
LWP failed with code[403] message[Browserblocked]
I think, only specific browsers are let through the proxy server, so I need to set the useragent for cpan. Where can I set it? I don't see anything similar in o conf.
Rewriting the code of site\lib\LWP\UserAgent.pm
sub _agent { "libwww-perl/$VERSION" }
say to:
sub _agent { 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0' }
solves the problem, but is this really the official solution?

perl: firefox rejects my cookie?

I have the following script at �://192.168.1.3/homeworks/hw10/testcookie.cgi:
#!/usr/bin/perl -wT
use CGI qw(:standard);
use CGI::Carp qw(warningsToBrowser fatalsToBrowser);
use CGI::Cookie qw(cookie);
use strict;
use warnings;
sub makecookie {
return my $cookie = cookie(-name=>'hw10',
-value=>shift,
-expires=>shift,
-path=>'/hw10/testcookie.cgi',
-domain=>'192.168.1.3',
-secure=>1);
}
my $cgi = CGI->new();
my $cookie = makecookie("192.168.1.3",'+3d');
print STDOUT $cgi->header(-cookie=>$cookie);
print STDOUT $cgi->start_html("Test Cookie");
print STDOUT "<h1>TEST</h1>";
print STDOUT $cgi->end_html();
I checked with Live headers that something was sent:
GET /homeworks/hw10/testcookie.cgi HTTP/1.1
Host: 192.168.1.3
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:34.0) Gecko/20100101 Firefox/34.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
HTTP/1.1 200 OK
Date: Wed, 24 Dec 2014 16:11:24 GMT
Server: Apache/2.2.22 (Debian)
Set-Cookie: hw10=192.168.1.3; domain=192.168.1.3; path=/hw10/testcookie.cgi; expires=Sat, 27-Dec-2014 16:11:24 GMT; secure
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 258
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=ISO-8859-1
----------------------------------------------------------
but firefox says there is no cookie associated with the site. Why does firefox reject the cookie?
The path component of the URL is
/homeworks/hw10/testcookie.cgi
yet you're trying to set a cookie for
/hw10/testcookie.cgi
The latter is not contained by the former, so that's an error. You might want to use
-path => $cgi->url( -absolute => 1 )
instead of
-path => '/hw10/testcookie.cgi'
This may not be the only error. You didn't provide the URL you requested, so I don't know if you're using HTTP or HTTPS. You'll have problems if you're using HTTP because you specified the cookie should only be provided over secure connections.

Setting all headers when using Powershell Invoke-WebRequest

I would like to download PDF file from online form. I used the fiddler to find out the header and post fields. However, I cannot get the header from Firefox to match with Powershell using Invoke-webrequest.
Original request
POST http://somesight/SPLeam/EAMDisplayReportWithParamValues.asp HTTP/1.1
Host: someserver
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:27.0) Gecko/20100101 Firefox/27.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://someserver/SPLeam/EAMPickParamValues.asp?plant=01&reportuser=reports&reportuserpw=pw&ReportName=SPLEAM%5CMNT+%2D+MM+Tracking+Rpt%2Erpt
Cookie: ASPSESDFSDIDSSQAQBQC=APHPHMNCHIECSFSSPOIKBDCAM`
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 132
Text0=12%2F1%2F2013&Text1=12%2F31%2F2013&Text2=MESGI&Drop2=MESCI&Text3=01&Text4=&Drop4=&Text5=R&Text6=N&cmdSaveParameters=Run+Report
POWERSHELL
POST http://someserver/SPLeam/EAMPickParamValues.asp?plant=01&reportuser=reports&reportuserpw=pw&ReportName=SPLEAM%5CMNT+%2D+MM+Tracking+Rpt%2Erpt HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT; Windows NT 6.1; en-US) WindowsPowerShell/3.0
Content-Type: application/x-www-form-urlencoded
Host: someserver
Content-Length: 129
Connection: Keep-Alive
cmdSaveParameters=Run%2bReport&Text3=01&Text5=R&Text1=12%252F31%252F2013&Drop2=MI&Text2=MI&Text4=&Text6=N&Text0=12%252F1%252F2013
I run it as
$f = invoke-webrequest -uri $url -UserAgent $useragent -method Post -Body $param
How can I make Powershell's request match what Firefox sent?
.NET's HTTPWebRequest object has separate properties for the Accept and Referer headers; you must manually set these properties instead of trying to add the headers directly.
(These header restrictions exist partly for security purposes; partial trust code cannot set the properties).