snort was alive, but now she's dead. no clue. :( - snort

I got snort up and running the other and since an attempt to get barnyard2 working, snort is no longer sniffing. She fires up error free, the port monitoring is still blasting traffic at her, no errors in the logs. But even a custom google access rule fails to fire off in the buffer. went from seeing everything to seeing nothing overnight and I didn't do anything but compiled barnyard2 with mysql support she last worked???
New server, new install.
#--------------------------------------------------
# VRT Rule Packages Snort.conf
#
# For more information visit us at:
# http://www.snort.org Snort Website
# http://vrt-blog.snort.org/ Sourcefire VRT Blog
#
# Mailing list Contact: snort-sigs#lists.sourceforge.net
# False Positive reports: fp#sourcefire.com
# Snort bugs: bugs#snort.org
#
# Compatible with Snort Versions:
# VERSIONS : 2.9.5.3
#
# Snort build options:
# OPTIONS : --enable-gre --enable-mpls --enable-targetbased --enable-ppm --enable-perfprofiling --enable-zlib --enable-active-response --enable-normalizer --enable-reload --enable-react --enable-flexresp3
#
# Additional information:
# This configuration file enables active response, to run snort in
# test mode -T you are required to supply an interface -i <interface>
# or test mode will fail to fully validate the configuration and
# exit with a FATAL error
#--------------------------------------------------
###################################################
# This file contains a sample snort configuration.
# You should take the following steps to create your own custom configuration:
#
# 1) Set the network variables.
# 2) Configure the decoder
# 3) Configure the base detection engine
# 4) Configure dynamic loaded libraries
# 5) Configure preprocessors
# 6) Configure output plugins
# 7) Customize your rule set
# 8) Customize preprocessor and decoder rule set
# 9) Customize shared object rule set
###################################################
###################################################
# Step #1: Set the network variables. For more information, see README.variables
###################################################
# Setup the network addresses you are protecting
ipvar HOME_NET any
# Set up the external network addresses. Leave as "any" in most situations
ipvar EXTERNAL_NET any
# List of DNS servers on your network
ipvar DNS_SERVERS $HOME_NET
# List of SMTP servers on your network
ipvar SMTP_SERVERS $HOME_NET
# List of web servers on your network
ipvar HTTP_SERVERS $HOME_NET
# List of sql servers on your network
ipvar SQL_SERVERS $HOME_NET
# List of telnet servers on your network
ipvar TELNET_SERVERS $HOME_NET
# List of ssh servers on your network
ipvar SSH_SERVERS $HOME_NET
# List of ftp servers on your network
ipvar FTP_SERVERS $HOME_NET
# List of sip servers on your network
ipvar SIP_SERVERS $HOME_NET
# List of ports you run web servers on
portvar HTTP_PORTS [80,81,82,83,84,85,86,87,88,89,90,311,383,591,593,631,901,1220,1414,1741,1830,2301,2381,2809,3037,3057,3128,3443,3702,4343,4848,5250,6080,6988,7000,7001,7144,7145,7510,7777,7779,8000,8008,8014,8028,8080,8085,8088,8090,8118,8123,8180,8181,8222,8243,8280,8300,8500,8800,8888,8899,9000,9060,9080,9090,9091,9443,9999,10000,11371,34443,34444,41080,50000,50002,55555]
# List of ports you want to look for SHELLCODE on.
portvar SHELLCODE_PORTS !80
# List of ports you might see oracle attacks on
portvar ORACLE_PORTS 1024:
# List of ports you want to look for SSH connections on:
portvar SSH_PORTS 22
# List of ports you run ftp servers on
portvar FTP_PORTS [21,2100,3535]
# List of ports you run SIP servers on
portvar SIP_PORTS [5060,5061,5600]
# List of file data ports for file inspection
portvar FILE_DATA_PORTS [$HTTP_PORTS,110,143]
# List of GTP ports for GTP preprocessor
portvar GTP_PORTS [2123,2152,3386]
# other variables, these should not be modified
ipvar AIM_SERVERS [64.12.24.0/23,64.12.28.0/23,64.12.161.0/24,64.12.163.0/24,64.12.200.0/24,205.188.3.0/24,205.188.5.0/24,205.188.7.0/24,205.188.9.0/24,205.188.153.0/24,205.188.179.0/24,205.188.248.0/24]
# Path to your rules files (this can be a relative path)
# Note for Windows users: You are advised to make this an absolute path,
# such as: c:\snort\rules
var RULE_PATH /etc/snort/rules/rules
var SO_RULE_PATH /etc/snort/rules/so_rules
var PREPROC_RULE_PATH /etc/snort/rules/preproc_rules
# If you are using reputation preprocessor set these
# Currently there is a bug with relative paths, they are relative to where snort is
# not relative to snort.conf like the above variables
# This is completely inconsistent with how other vars work, BUG 89986
# Set the absolute path appropriately
var WHITE_LIST_PATH /etc/snort/rules/rules
var BLACK_LIST_PATH /etc/snort/rules/rules
###################################################
# Step #2: Configure the decoder. For more information, see README.decode
###################################################
# Stop generic decode events:
config disable_decode_alerts
# Stop Alerts on experimental TCP options
config disable_tcpopt_experimental_alerts
# Stop Alerts on obsolete TCP options
config disable_tcpopt_obsolete_alerts
# Stop Alerts on T/TCP alerts
config disable_tcpopt_ttcp_alerts
# Stop Alerts on all other TCPOption type events:
config disable_tcpopt_alerts
# Stop Alerts on invalid ip options
config disable_ipopt_alerts
# Alert if value in length field (IP, TCP, UDP) is greater th elength of the packet
# config enable_decode_oversized_alerts
# Same as above, but drop packet if in Inline mode (requires enable_decode_oversized_alerts)
# config enable_decode_oversized_drops
# Configure IP / TCP checksum mode
config checksum_mode: all
# Configure maximum number of flowbit references. For more information, see README.flowbits
# config flowbits_size: 64
# Configure ports to ignore
# config ignore_ports: tcp 21 6667:6671 1356
# config ignore_ports: udp 1:17 53
# Configure active response for non inline operation. For more information, see REAMDE.active
# config response: eth0 attempts 2
# Configure DAQ related options for inline operation. For more information, see README.daq
#
# config daq: <type>
# config daq_dir: <dir>
# config daq_mode: <mode>
# config daq_var: <var>
#
# <type> ::= pcap | afpacket | dump | nfq | ipq | ipfw
# <mode> ::= read-file | passive | inline
# <var> ::= arbitrary <name>=<value passed to DAQ
# <dir> ::= path as to where to look for DAQ module so's
# Configure specific UID and GID to run snort as after dropping privs. For more information see snort -h command line options
#
# config set_gid:
# config set_uid:
# Configure default snaplen. Snort defaults to MTU of in use interface. For more information see README
#
# config snaplen:
#
# Configure default bpf_file to use for filtering what traffic reaches snort. For more information see snort -h command line options (-F)
#
# config bpf_file:
#
# Configure default log directory for snort to log to. For more information see snort -h command line options (-l)
#
# config logdir:
###################################################
# Step #3: Configure the base detection engine. For more information, see README.decode
###################################################
# Configure PCRE match limitations
config pcre_match_limit: 3500
config pcre_match_limit_recursion: 1500
# Configure the detection engine See the Snort Manual, Configuring Snort - Includes - Config
config detection: search-method ac-split search-optimize max-pattern-len 20
# Configure the event queue. For more information, see README.event_queue
config event_queue: max_queue 8 log 5 order_events content_length
###################################################
## Configure GTP if it is to be used.
## For more information, see README.GTP
####################################################
# config enable_gtp
###################################################
# Per packet and rule latency enforcement
# For more information see README.ppm
###################################################
# Per Packet latency configuration
#config ppm: max-pkt-time 250, \
# fastpath-expensive-packets, \
# pkt-log
# Per Rule latency configuration
#config ppm: max-rule-time 200, \
# threshold 3, \
# suspend-expensive-rules, \
# suspend-timeout 20, \
# rule-log alert
###################################################
# Configure Perf Profiling for debugging
# For more information see README.PerfProfiling
###################################################
#config profile_rules: print all, sort avg_ticks
#config profile_preprocs: print all, sort avg_ticks
###################################################
# Configure protocol aware flushing
# For more information see README.stream5
###################################################
config paf_max: 16000
###################################################
# Step #4: Configure dynamic loaded libraries.
# For more information, see Snort Manual, Configuring Snort - Dynamic Modules
###################################################
# path to dynamic preprocessor libraries
dynamicpreprocessor directory /usr/local/lib/snort_dynamicpreprocessor/
# path to base preprocessor engine
dynamicengine /usr/local/lib/snort_dynamicengine/libsf_engine.so
# path to dynamic rules libraries
dynamicdetection directory /usr/local/lib/snort_dynamicrules
###################################################
# Step #5: Configure preprocessors
# For more information, see the Snort Manual, Configuring Snort - Preprocessors
###################################################
# GTP Control Channle Preprocessor. For more information, see README.GTP
# preprocessor gtp: ports { 2123 3386 2152 }
# Inline packet normalization. For more information, see README.normalize
# Does nothing in IDS mode
preprocessor normalize_ip4
preprocessor normalize_tcp: ips ecn stream
preprocessor normalize_icmp4
preprocessor normalize_ip6
preprocessor normalize_icmp6
# Target-based IP defragmentation. For more inforation, see README.frag3
preprocessor frag3_global: max_frags 65536
preprocessor frag3_engine: policy windows detect_anomalies overlap_limit 10 min_fragment_length 100 timeout 180
# Target-Based stateful inspection/stream reassembly. For more inforation, see README.stream5
preprocessor stream5_global: track_tcp yes, \
track_udp yes, \
track_icmp no, \
max_tcp 262144, \
max_udp 131072, \
max_active_responses 2, \
min_response_seconds 5
preprocessor stream5_tcp: policy windows, detect_anomalies, require_3whs 180, \
overlap_limit 10, small_segments 3 bytes 150, timeout 180, \
ports client 21 22 23 25 42 53 70 79 109 110 111 113 119 135 136 137 139 143 \
161 445 513 514 587 593 691 1433 1521 1741 2100 3306 6070 6665 6666 6667 6668 6669 \
7000 8181 32770 32771 32772 32773 32774 32775 32776 32777 32778 32779, \
ports both 80 81 82 83 84 85 86 87 88 89 90 110 311 383 443 465 563 591 593 631 636 901 989 992 993 994 995 1220 1414 1830 2301 2381 2809 3037 3057 3128 3443 3702 4343 4848 5250 6080 6988 7907 7000 7001 7144 7145 7510 7802 7777 7779 \
7801 7900 7901 7902 7903 7904 7905 7906 7908 7909 7910 7911 7912 7913 7914 7915 7916 \
7917 7918 7919 7920 8000 8008 8014 8028 8080 8085 8088 8090 8118 8123 8180 8222 8243 8280 8300 8500 8800 8888 8899 9000 9060 9080 9090 9091 9443 9999 10000 11371 34443 34444 41080 50000 50002 55555
preprocessor stream5_udp: timeout 180
# performance statistics. For more information, see the Snort Manual, Configuring Snort - Preprocessors - Performance Monitor
# preprocessor perfmonitor: time 300 file /var/snort/snort.stats pktcnt 10000
# HTTP normalization and anomaly detection. For more information, see README.http_inspect
preprocessor http_inspect: global iis_unicode_map unicode.map 1252 compress_depth 65535 decompress_depth 65535
preprocessor http_inspect_server: server default \
http_methods { GET POST PUT SEARCH MKCOL COPY MOVE LOCK UNLOCK NOTIFY POLL BCOPY BDELETE BMOVE LINK UNLINK OPTIONS HEAD DELETE TRACE TRACK CONNECT SOURCE SUBSCRIBE UNSUBSCRIBE PROPFIND PROPPATCH BPROPFIND BPROPPATCH RPC_CONNECT PROXY_SUCCESS BITS_POST CCM_POST SMS_POST RPC_IN_DATA RPC_OUT_DATA RPC_ECHO_DATA } \
chunk_length 500000 \
server_flow_depth 0 \
client_flow_depth 0 \
post_depth 65495 \
oversize_dir_length 500 \
max_header_length 750 \
max_headers 100 \
max_spaces 200 \
small_chunk_length { 10 5 } \
ports { 80 81 82 83 84 85 86 87 88 89 90 311 383 591 593 631 901 1220 1414 1741 1830 2301 2381 2809 3037 3057 3128 3443 3702 4343 4848 5250 6080 6988 7000 7001 7144 7145 7510 7777 7779 8000 8008 8014 8028 8080 8085 8088 8090 8118 8123 8180 8181 8222 8243 8280 8300 8500 8800 8888 8899 9000 9060 9080 9090 9091 9443 9999 10000 11371 34443 34444 41080 50000 50002 55555 } \
non_rfc_char { 0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07 } \
enable_cookie \
extended_response_inspection \
inspect_gzip \
normalize_utf \
unlimited_decompress \
normalize_javascript \
apache_whitespace no \
ascii no \
bare_byte no \
directory no \
double_decode no \
iis_backslash no \
iis_delimiter no \
iis_unicode no \
multi_slash no \
utf_8 no \
u_encode yes \
webroot no
# ONC-RPC normalization and anomaly detection. For more information, see the Snort Manual, Configuring Snort - Preprocessors - RPC Decode
preprocessor rpc_decode: 111 32770 32771 32772 32773 32774 32775 32776 32777 32778 32779 no_alert_multiple_requests no_alert_large_fragments no_alert_incomplete
# Back Orifice detection.
preprocessor bo
# FTP / Telnet normalization and anomaly detection. For more information, see README.ftptelnet
preprocessor ftp_telnet: global inspection_type stateful encrypted_traffic no check_encrypted
preprocessor ftp_telnet_protocol: telnet \
ayt_attack_thresh 20 \
normalize ports { 23 } \
detect_anomalies
preprocessor ftp_telnet_protocol: ftp server default \
def_max_param_len 100 \
ports { 21 2100 3535 } \
telnet_cmds yes \
ignore_telnet_erase_cmds yes \
ftp_cmds { ABOR ACCT ADAT ALLO APPE AUTH CCC CDUP } \
ftp_cmds { CEL CLNT CMD CONF CWD DELE ENC EPRT } \
ftp_cmds { EPSV ESTA ESTP FEAT HELP LANG LIST LPRT } \
ftp_cmds { LPSV MACB MAIL MDTM MIC MKD MLSD MLST } \
ftp_cmds { MODE NLST NOOP OPTS PASS PASV PBSZ PORT } \
ftp_cmds { PROT PWD QUIT REIN REST RETR RMD RNFR } \
ftp_cmds { RNTO SDUP SITE SIZE SMNT STAT STOR STOU } \
ftp_cmds { STRU SYST TEST TYPE USER XCUP XCRC XCWD } \
ftp_cmds { XMAS XMD5 XMKD XPWD XRCP XRMD XRSQ XSEM } \
ftp_cmds { XSEN XSHA1 XSHA256 } \
alt_max_param_len 0 { ABOR CCC CDUP ESTA FEAT LPSV NOOP PASV PWD QUIT REIN STOU SYST XCUP XPWD } \
alt_max_param_len 200 { ALLO APPE CMD HELP NLST RETR RNFR STOR STOU XMKD } \
alt_max_param_len 256 { CWD RNTO } \
alt_max_param_len 400 { PORT } \
alt_max_param_len 512 { SIZE } \
chk_str_fmt { ACCT ADAT ALLO APPE AUTH CEL CLNT CMD } \
chk_str_fmt { CONF CWD DELE ENC EPRT EPSV ESTP HELP } \
chk_str_fmt { LANG LIST LPRT MACB MAIL MDTM MIC MKD } \
chk_str_fmt { MLSD MLST MODE NLST OPTS PASS PBSZ PORT } \
chk_str_fmt { PROT REST RETR RMD RNFR RNTO SDUP SITE } \
chk_str_fmt { SIZE SMNT STAT STOR STRU TEST TYPE USER } \
chk_str_fmt { XCRC XCWD XMAS XMD5 XMKD XRCP XRMD XRSQ } \
chk_str_fmt { XSEM XSEN XSHA1 XSHA256 } \
cmd_validity ALLO < int [ char R int ] > \
cmd_validity EPSV < [ { char 12 | char A char L char L } ] > \
cmd_validity MACB < string > \
cmd_validity MDTM < [ date nnnnnnnnnnnnnn[.n[n[n]]] ] string > \
cmd_validity MODE < char ASBCZ > \
cmd_validity PORT < host_port > \
cmd_validity PROT < char CSEP > \
cmd_validity STRU < char FRPO [ string ] > \
cmd_validity TYPE < { char AE [ char NTC ] | char I | char L [ number ] } >
preprocessor ftp_telnet_protocol: ftp client default \
max_resp_len 256 \
bounce yes \
ignore_telnet_erase_cmds yes \
telnet_cmds yes
# SMTP normalization and anomaly detection. For more information, see README.SMTP
preprocessor smtp: ports { 25 465 587 691 } \
inspection_type stateful \
b64_decode_depth 0 \
qp_decode_depth 0 \
bitenc_decode_depth 0 \
uu_decode_depth 0 \
log_mailfrom \
log_rcptto \
log_filename \
log_email_hdrs \
normalize cmds \
normalize_cmds { ATRN AUTH BDAT CHUNKING DATA DEBUG EHLO EMAL ESAM ESND ESOM ETRN EVFY } \
normalize_cmds { EXPN HELO HELP IDENT MAIL NOOP ONEX QUEU QUIT RCPT RSET SAML SEND SOML } \
normalize_cmds { STARTTLS TICK TIME TURN TURNME VERB VRFY X-ADAT X-DRCP X-ERCP X-EXCH50 } \
normalize_cmds { X-EXPS X-LINK2STATE XADR XAUTH XCIR XEXCH50 XGEN XLICENSE XQUE XSTA XTRN XUSR } \
max_command_line_len 512 \
max_header_line_len 1000 \
max_response_line_len 512 \
alt_max_command_line_len 260 { MAIL } \
alt_max_command_line_len 300 { RCPT } \
alt_max_command_line_len 500 { HELP HELO ETRN EHLO } \
alt_max_command_line_len 255 { EXPN VRFY ATRN SIZE BDAT DEBUG EMAL ESAM ESND ESOM EVFY IDENT NOOP RSET } \
alt_max_command_line_len 246 { SEND SAML SOML AUTH TURN ETRN DATA RSET QUIT ONEX QUEU STARTTLS TICK TIME TURNME VERB X-EXPS X-LINK2STATE XADR XAUTH XCIR XEXCH50 XGEN XLICENSE XQUE XSTA XTRN XUSR } \
valid_cmds { ATRN AUTH BDAT CHUNKING DATA DEBUG EHLO EMAL ESAM ESND ESOM ETRN EVFY } \
valid_cmds { EXPN HELO HELP IDENT MAIL NOOP ONEX QUEU QUIT RCPT RSET SAML SEND SOML } \
valid_cmds { STARTTLS TICK TIME TURN TURNME VERB VRFY X-ADAT X-DRCP X-ERCP X-EXCH50 } \
valid_cmds { X-EXPS X-LINK2STATE XADR XAUTH XCIR XEXCH50 XGEN XLICENSE XQUE XSTA XTRN XUSR } \
xlink2state { enabled }
# Portscan detection. For more information, see README.sfportscan
# preprocessor sfportscan: proto { all } memcap { 10000000 } sense_level { low }
# ARP spoof detection. For more information, see the Snort Manual - Configuring Snort - Preprocessors - ARP Spoof Preprocessor
# preprocessor arpspoof
# preprocessor arpspoof_detect_host: 192.168.40.1 f0:0f:00:f0:0f:00
# SSH anomaly detection. For more information, see README.ssh
preprocessor ssh: server_ports { 22 } \
autodetect \
max_client_bytes 19600 \
max_encrypted_packets 20 \
max_server_version_len 100 \
enable_respoverflow enable_ssh1crc32 \
enable_srvoverflow enable_protomismatch
# SMB / DCE-RPC normalization and anomaly detection. For more information, see README.dcerpc2
preprocessor dcerpc2: memcap 102400, events [co ]
preprocessor dcerpc2_server: default, policy WinXP, \
detect [smb [139,445], tcp 135, udp 135, rpc-over-http-server 593], \
autodetect [tcp 1025:, udp 1025:, rpc-over-http-server 1025:], \
smb_max_chain 3, smb_invalid_shares ["C$", "D$", "ADMIN$"]
# DNS anomaly detection. For more information, see README.dns
preprocessor dns: ports { 53 } enable_rdata_overflow
# SSL anomaly detection and traffic bypass. For more information, see README.ssl
preprocessor ssl: ports { 443 465 563 636 989 992 993 994 995 7801 7802 7900 7901 7902 7903 7904 7905 7906 7907 7908 7909 7910 7911 7912 7913 7914 7915 7916 7917 7918 7919 7920 }, trustservers, noinspect_encrypted
# SDF sensitive data preprocessor. For more information see README.sensitive_data
preprocessor sensitive_data: alert_threshold 25
# SIP Session Initiation Protocol preprocessor. For more information see README.sip
preprocessor sip: max_sessions 40000, \
ports { 5060 5061 5600 }, \
methods { invite \
cancel \
ack \
bye \
register \
options \
refer \
subscribe \
update \
join \
info \
message \
notify \
benotify \
do \
qauth \
sprack \
publish \
service \
unsubscribe \
prack }, \
max_uri_len 512, \
max_call_id_len 80, \
max_requestName_len 20, \
max_from_len 256, \
max_to_len 256, \
max_via_len 1024, \
max_contact_len 512, \
max_content_len 2048
# IMAP preprocessor. For more information see README.imap
preprocessor imap: \
ports { 143 } \
b64_decode_depth 0 \
qp_decode_depth 0 \
bitenc_decode_depth 0 \
uu_decode_depth 0
# POP preprocessor. For more information see README.pop
preprocessor pop: \
ports { 110 } \
b64_decode_depth 0 \
qp_decode_depth 0 \
bitenc_decode_depth 0 \
uu_decode_depth 0
# Modbus preprocessor. For more information see README.modbus
preprocessor modbus: ports { 502 }
# DNP3 preprocessor. For more information see README.dnp3
preprocessor dnp3: ports { 20000 } \
memcap 262144 \
check_crc
# Reputation preprocessor. For more information see README.reputation
preprocessor reputation: \
memcap 500, \
priority whitelist, \
nested_ip inner, \
whitelist $WHITE_LIST_PATH/white_list.rules, \
blacklist $BLACK_LIST_PATH/black_list.rules
###################################################
# Step #6: Configure output plugins
# For more information, see Snort Manual, Configuring Snort - Output Modules
###################################################
# unified2
# Recommended for most installs
output unified2: filename merged.log, limit 128, nostamp, mpls_event_types, vlan_event_types
# Additional configuration for specific types of installs
# output alert_unified2: filename snort.alert, limit 128, nostamp
# output log_unified2: filename snort.log, limit 128, nostamp
# syslog
# output alert_syslog: LOG_AUTH LOG_ALERT
# pcap
# #output log_tcpdump: tcpdump.log
# metadata reference data. do not modify these lines
include classification.config
include reference.config
Had to chop due to size restrictions.
My complete snort.conf here -> http://www.bpaste.net/show/K4IoLi86w5fdsoRweQ0m/
My complete startup here -> http://www.bpaste.net/show/uOVVEgNWHFYTwZNmBezI/

internal network issues were the cause

Related

How to grep or awk through ghe-logs-tail?

Github Enterprise Server comes with a utility:
ghe-logs-tail
this tails all GHE Server logs simultaneously and prints the flow to the console for the user to view.
When trying to grep through these through, eg. for the "error" string like so:
ghe-logs-tail | grep --line-buffered -i "error"
The console will not print the flow and instead includes an error that some log files could not be opened for some reason.
$ ghe-logs-tail | grep --line-buffered -i "error"
/var/log/github/audit.log /var/log/github/auth.log /var/log/github/exceptions.log /var/log/github/gitauth.log /var/log/github/personal.log /var/log/github/production.log /var/log/github/resqued.log /var/log/github/unicorn.log /var/log/enterprise-manage/login_attempts.log /var/log/enterprise-manage/unicorn.log /var/log/github/auth.log /var/log/auth.log /var/log/nginx/alambic.assets.error.log /var/log/nginx/alambic.assets.log /var/log/nginx/alambic.avatars.error.log /var/log/nginx/alambic.avatars.log /var/log/nginx/alambic.error.log /var/log/nginx/alambic.log /var/log/nginx/avatars.error.log /var/log/nginx/avatars.log /var/log/nginx/credits.error.log /var/log/nginx/credits.log /var/log/nginx/enterprise-manage.error.log /var/log/nginx/enterprise-manage.log /var/log/nginx/error.log /var/log/nginx/gist.error.log /var/log/nginx/gist.log /var/log/nginx/github.error.log /var/log/nginx/github.log /var/log/nginx/pages.error.log /var/log/nginx/pages.log /var/log/nginx/raw.error.log /var/log/nginx/raw.log /var/log/nginx/render.error.log /var/log/nginx/render.log /var/log/nginx/static-maintenance.error.log /var/log/nginx/static-maintenance.log /var/log/nginx/storage.error.log /var/log/nginx/storage.log /data/user/common/ghe-config.log /var/log/syslog /var/log/dmesg /var/log/mysql/*.log /var/log/redis/redis.log /var/log/haproxy.log
==> /var/log/nginx/alambic.assets.error.log <==
==> /var/log/nginx/alambic.avatars.error.log <==
==> /var/log/nginx/alambic.error.log <==
==> /var/log/nginx/avatars.error.log <==
==> /var/log/nginx/credits.error.log <==
==> /var/log/nginx/enterprise-manage.error.log <==
==> /var/log/nginx/error.log <==
==> /var/log/nginx/gist.error.log <==
==> /var/log/nginx/github.error.log <==
tail: cannot open '/var/log/mysql/*.log' for reading==> /var/log/nginx/pages.error.log <==
==> /var/log/nginx/raw.error.log <==
: No such file or directory==> /var/log/nginx/render.error.log <==
==> /var/log/nginx/static-maintenance.error.log <==
==> /var/log/nginx/storage.error.log <==
tail: cannot open '/var/log/redis/redis.log' for reading: No such file or directory
What is the correct way to do this with a native tool like grep/awk?
I would obviously like to have all lines that contain the matching string printed out in real time to the console.
you can filter out grep errors by routing them to the null device (blackhole)
... | grep ... 2>/dev/null

Galera connection issues over haproxy

In our K8 cluster, we use haproxy app for connecting to Galera cluster.
Our haproxy.cnf file looks like
global
maxconn 2048
external-check
stats socket /var/run/haproxy.sock mode 600 expose-fd listeners level user
user haproxy
group haproxy
defaults
log global
mode tcp
retries 10
timeout client 30000
timeout connect 100500
timeout server 30000
frontend mysql-router-service
bind *:6446
mode tcp
option tcplog
default_backend galera_cluster_backend
# MySQL Cluster BE configuration
backend galera_cluster_backend
mode tcp
option tcpka
option mysql-check user haproxy
balance source
server pitipana-opsdb1 192.168.144.82:3306 check weight 1
server pitipana-opsdb2 192.168.144.83:3306 check weight 1
server pitipana-opsdb3 192.168.144.84:3306 check weight 1
Dockerfile for creating haproxy image
FROM haproxy:2.3
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
In my Galera nodes, I get constant warning in /var/log/mysql/error.log
2021-12-20 21:16:47 5942 [Warning] Aborted connection 5942 to db: 'ourdb' user: 'ouruser' host: '192.168.1.2' (Got an error reading communication packets)
2021-12-20 21:16:47 5943 [Warning] Aborted connection 5943 to db: 'ourdb' user: 'ouruser' host: '192.168.1.2' (Got an error reading communication packets)
2021-12-20 21:16:47 5944 [Warning] Aborted connection 5944 to db: 'ourdb' user: 'ouruser' host: '192.168.1.2' (Got an error reading communication packets)
I had increased max_packet_size to 64MB and max_connections to 1000.
When I take a tcpdump from galera node :
Frame 16: 106 bytes on wire (848 bits), 106 bytes captured (848 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 192.168.1.2, Dst: 192.168.10.3
Transmission Control Protocol, Src Port: 62495, Dst Port: 3306, Seq: 1, Ack: 1, Len: 50
Source Port: 62495
Destination Port: 3306
[Stream index: 2]
[TCP Segment Len: 50]
Sequence number: 1 (relative sequence number)
[Next sequence number: 51 (relative sequence number)]
Acknowledgment number: 1 (relative ack number)
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...1 .... = Acknowledgment: Set
.... .... 1... = Push: Set
.... .... .0.. = Reset: Not set
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
[TCP Flags: ·······AP···]
Window size value: 507
[Calculated window size: 64896]
[Window size scaling factor: 128]
Checksum: 0x3cec [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
[SEQ/ACK analysis]
[Timestamps]
TCP payload (50 bytes)
[PDU Size: 45]
[PDU Size: 5]
MySQL Protocol
Packet Length: 41
Packet Number: 1
Request Command SLEEP
Command: SLEEP (0)
Payload: 820000008000012100000000000000000000000000000000...
[Expert Info (Warning/Protocol): Unknown/invalid command code]
[Unknown/invalid command code]
[Severity level: Warning]
[Group: Protocol]
MySQL Protocol
Packet Length: 1
Packet Number: 0
Request Command Quit
Command: Quit (1)
Here 192.168.1.2 is a K8 worker node and 192.168.10.3 is the galera node.
When I connect our applications in K8, we can access to applications, but when we try to edit, we get stuck.
Any suggestion to fix this?

How to get the request sent by my computer using tshark

I ran tshark -V > file.log in my terminal and then in a separate terminal I ran curl 'www.google.com'. I then returned to the first terminal, shut off tshark and then looked at file.log. In it there are a number of 'frames'. For example, here is one of them:
Frame 42: 66 bytes on wire (528 bits), 66 bytes captured (528 bits) on interface en0, id 0
Interface id: 0 (en0)
Interface name: en0
Interface description: Wi-Fi
Encapsulation type: Ethernet (1)
Arrival Time: Nov 3, 2020 17:28:15.022217000 PST
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1604453295.022217000 seconds
[Time delta from previous captured frame: 0.000244000 seconds]
[Time delta from previous displayed frame: 0.000244000 seconds]
[Time since reference or first frame: 12.164253000 seconds]
Frame Number: 42
Frame Length: 66 bytes (528 bits)
Capture Length: 66 bytes (528 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ethertype:ip:tcp]
Ethernet II, Src: *****, Dst: *****
Destination: *****
Address: *****
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Source: Apple_81:82:2f (ac:bc:32:81:82:2f)
Address: Apple_81:82:2f (ac:bc:32:81:82:2f)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: *****, Dst: *****
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
0000 00.. = Differentiated Services Codepoint: Default (0)
.... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
Total Length: 52
Identification: 0x0000 (0)
Flags: 0x40, Don't fragment
0... .... = Reserved bit: Not set
.1.. .... = Don't fragment: Set
..0. .... = More fragments: Not set
Fragment Offset: 0
Time to Live: 64
Protocol: TCP (6)
Header Checksum: 0xabe8 [validation disabled]
[Header checksum status: Unverified]
Source Address: 134.87.182.156
Destination Address: 172.217.165.14
Transmission Control Protocol, Src Port: 55888, Dst Port: 80, Seq: 76, Ack: 530, Len: 0
Source Port: 55888
Destination Port: 80
[Stream index: 3]
[TCP Segment Len: 0]
Sequence Number: 76 (relative sequence number)
Sequence Number (raw): 2379446728
[Next Sequence Number: 76 (relative sequence number)]
Acknowledgment Number: 530 (relative ack number)
Acknowledgment number (raw): 278173922
1000 .... = Header Length: 32 bytes (8)
Flags: 0x010 (ACK)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...1 .... = Acknowledgment: Set
.... .... 0... = Push: Not set
.... .... .0.. = Reset: Not set
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
[TCP Flags: ·······A····]
Window: 2048
[Calculated window size: 131072]
[Window size scaling factor: 64]
Checksum: 0xc5b6 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
TCP Option - No-Operation (NOP)
Kind: No-Operation (1)
TCP Option - No-Operation (NOP)
Kind: No-Operation (1)
TCP Option - Timestamps: TSval 190086814, TSecr 4203481592
Kind: Time Stamp Option (8)
Length: 10
Timestamp value: 190086814
Timestamp echo reply: 4203481592
[SEQ/ACK analysis]
[This is an ACK to the segment in frame: 41]
[The RTT to ACK the segment was: 0.000244000 seconds]
[iRTT: 0.064637000 seconds]
[Timestamps]
[Time since first frame in this TCP stream: 0.219685000 seconds]
[Time since previous frame in this TCP stream: 0.000244000 seconds]
I think each frame corresponds to a single request sent or received by my computer. I want to know how to reconstruct the exact request sent by my computer to googles server. Additionally, I want to know how to capture whatever was returned by the server.
During the packet capture you can use the -f (capture filter) option to extract only the packets linked to www.google.com
tshark -a duration:10 -T text -V -f "host www.google.com" > capture.txt
I set the -a (autostop) capture option to 10 seconds, because I was only doing the curl 'www.google.com' once.
The command above will capture all TCP and UDP related to the curl request.
If you want to reassemble the connection after a capture you need to create a pcap file during the capture:
# this is capturing all traffic
tshark -a duration:10 -w capture.pcap
You can query this pcap file in multiple ways:
tshark -r capture.pcap -Y http.request -T fields -e http.host -e http.user_agent
# output
www.google.com curl/7.64.1'
tshark -r capture.pcap -Y "(http.host == www.google.com)"
44 1.631029 192.168.86.35 → 64.233.177.105 HTTP 144 GET / HTTP/1.1
tshark -r capture.pcap -Y "dns.qry.name == www.google.com"
#output
36 1.546800 192.168.86.35 → 192.168.86.1 DNS 74 Standard query 0xe159 A www.google.com
39 1.587633 192.168.86.1 → 192.168.86.35 DNS 170 Standard query response 0xe159 A www.google.com A 64.233.177.105 A 64.233.177.104 A 64.233.177.103 A 64.233.177.99 A 64.233.177.147 A 64.233.177.106
# get the frame details for TCP packets associated with google
tshark -r capture.pcap -T fields -e tcp.stream -Y "tcp contains google"
#output frame number(s)
# get the frame details for UDP packets associated with google
tshark -r capture.pcap -T fields -e udp.stream -Y "udp contains google"
# follow the TCP frame
# this should return the curl request for www.google.com
tshark -r capture.pcap -q -z follow,tcp,ascii,frame_number
# follow the UDP frame
tshark -r capture.pcap -q -z follow,udp,ascii,frame_number
Hopefully, this answer helps solve your question.

Perlbrew fails to install perl-5.30.0, op/groups.t at fault?

I'm attempting to install perl 5.30.0 from perl 5.16.3, I run
perlbrew install perl-5.30.0
but the installation fails, even with -f,
and selections from the error file show:
~/perl5/perlbrew/build/perl-5.30.0/perl-5.30.0$ grep -A1 -B1 -i failed /home/703404669/perl5/perlbrew/build.perl-5.30.0.log
opmini.o perlmini.o gv.o toke.o perly.o pad.o regcomp.o dump.o util.o mg.o reentr.o mro_core.o keywords.o hv.o av.o run.o pp_hot.o sv.o pp.o scope.o pp_ctl.o pp_sys.o doop.o doio.o regexec.o utf8.o taint.o deb.o universal.o globals.o perlio.o perlapi.o numeric.o mathoms.o locale.o pp_pack.o pp_sort.o caretx.o dquote.o time64.o miniperlmain.o -lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
./miniperl -w -Ilib -Idist/Exporter/lib -MExporter -e '<?>' || sh -c 'echo >&2 Failed to build miniperl. Please run make minitest; exit 1'
./miniperl -Ilib -f write_buildcustomize.pl
--
op/grep.t .......................................................... ok
# Failed test 1 - perl's `$(' agrees with `id -a 2>/dev/null || id 2>/dev/null' at op/groups.t line 117
op/groups.t ........................................................
Failed 1/2 subtests
op/gv.t ............................................................ ok
--
-------------------
op/groups.t (Wstat: 0 Tests: 2 Failed: 1)
Failed test: 1
Files=2656, Tests=1219021, 1082 wallclock secs (100.74 usr 14.39 sys + 712.86 cusr 87.83 csys = 915.82 CPU)
--
make: *** [test_harness] Error 1
##### Brew Failed #####
I can't find a solution on Google searches, I've found something similar on https://rt.perl.org/Public/Bug/Display.html?id=116775 but I don't see the solution, and I'm not even sure if that person had the same problem.
How can I get perl-5.30.0 installed on my perlbrew?
~/perl5/perlbrew/build/perl-5.30.0/perl-5.30.0$ id -a
uid=1227493(703404669) gid=60513(domain users) groups=60513(domain users),10(wheel),2000(bioinfom),2002(docker),3001(BUILTIN\users),3002(geno studies),3003(clm_users),3004(shrsf_all_employees),3005(svh_owa_ext),3006(svh_owa_non),3007(citrix access gateway),3008(citrix - ms outlook 2010),94696(citrix access gateway),94773(clm_users),95046(geno studies),97178(svh_owa_ext),97179(svh_owa_non),137580(shrsf_all_employees),529018(citrix - ms outlook 2010),562127(sp lab clsi link visitors),564451(fgo_owa_non),576177(shrsf_esbc_biobank),576507(sioux falls region users),615672(fargo remote e-mail),637092(sf remote access audit group),639414(fgo remote access audit group),642499(all emps_all regions),656741(all sites_sioux falls_sd),657700(all emps_system_all regions),657701(all emps_system_sf region),659636(all emps_sf region),667588,669081(polycom-cantonhosp-cart1-author),677540(room-sf-surgicaltower dakotafoodcourttiledarea-author),677669(rtls_standard),677823(room-sf-surgicaltower g734residencyclassroom-reviewer),703542(benefits_all_sf region),703546(benefits_emps_sf region),709223(benefits_all_all regions),709224(benefits_emps_all regions),716197(remote access audit cag),716198(remote access audit owa),716199(remote access audit vpn),719364(room-sf-stevenscenter publicaffairs),719474(equipment-sf-sanfordcenter-intlclinicsconfline-reviewers),722319(all emps_all regions_&_affiliates),725674(ent mobile connect),736131(ent lync users),760563(policies_sioux falls),796245(all emps_all regions_nonnested),797952(imagenetics - sioux falls),889435(one chart remote ent),889439(workflow tracer ent),1021861(avaya one-x communicator - sf),1037894(onerpt-admin-general),1112173(ent asa allowed),1127635(hrcommunications),1137950(ent remote connect),1137980(sioux falls users),1173104(all emps_all regions_non union),1192175(fgo owa groups),1206723(all emps_all regions_annual increase),1225668(ent symantec encryption users),1227493(703404669),1232214(avaya one-x communicator - sf-sp12),1236548(sccm_app_user_workflow tracer - mtn time - ltsr),1238714(sccm_app_user_workflow tracer - ltsr),1240689(sccm_app_user_one chart remote - ltsr),1242875 context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
~/perl5/perlbrew/build/perl-5.30.0/perl-5.30.0$ ./perl -le'print $('
60513 10 2000 2002 3001 3002 3003 3004 3005 3006 3007 3008 60513 94696 94773 95046 97178 97179 137580 529018 562127 564451 576177 576507 615672 637092 639414 642499 656741 657700 657701 659636 667588 669081 677540 677669 677823 703542 703546 709223 709224 716197 716198 716199 719364 719474 722319 725674 736131 760563 796245 797952 889435 889439 1021861 1037894 1112173 1127635 1137950 1137980 1173104 1192175 1206723 1225668 1227493 1232214 1236548 1238714 1240689 1242875

How to locate new certification using Certutil?

I need to create a new cert and then locate it to obtain the serial number (to use it later to get the public and private keys).
If I make a new certificate with makecert utility from Microsoft SDK's like this:
makecert -r -pe -a sha1 -n "CN=ElektroSoft" -b 01/01/2013 -e 01/01/2050 -ss my -$ individual
Then I suppose that cert will be stored in "my"...
..But when I try to list the "my" certificates I only get one entry, and is not my new certificate:
C:\>CERTUTIL -store my
my
================ Certificado 0 ================
Número de serie: 586a7358ebdce8854def26875f0f38ab
Emisor: CN=localhost
NotBefore: 22/02/2013 4:43
NotAfter: 22/02/2018 2:00
Sujeto: CN=localhost
La firma coincide con la clave pública
Certificado raíz: el sujeto coincide con el emisor
Plantilla:
Hash de cert(sha1): 1b 92 19 ef 19 ce d1 09 ad 87 13 73 56 0c cf 0a 57 29 cf 81
Contenedor de claves = IIS Express Development Certificate Container
Nombre de contenedor exclusivo: fad662b360941f26a1193357aab3c12d_a61f2a11-eaf9
-4c14-9a63-d3613bf3bd2c
Proveedor = Microsoft RSA SChannel Cryptographic Provider
Prueba de cifrado correcta
CertUtil: -store comando completado correctamente.
Then where is stored my new certificate?
Finally I did my code:
#Echo OFF
Setlocal enabledelayedexpansion
Set "Name=ElektroSoft"
Set "Password=Elektro"
Set "InitialDate=01/01/2013"
Set "ExpireDate=01/01/2050"
Set "SerialNumber=%Random%%Random%%Random%%Random%%Random%"
:: Make certificate
makecert -r -pe -a sha1 -n "CN=%Name%" -b "%InitialDate%" -e "%ExpireDate%" -$ individual -sr LocalMachine -ss my -cy authority -# "%SerialNumber%" "%Name%.cer"
:: Export certificate
For /F "Tokens=2 delims=:" %%# in ('certutil -store my') DO (
if "%%#" NEQ " CN=%Name%" (Set "SerialNumberHash=%%#") ELSE (
certutil -exportPFX -p "%Password%" my "!SerialNumberHash: =!" "%Name%.pfx"
Pause&Exit))
:: Check certificate
REM certutil -store my
:: Delete certificate
REM certutil -delstore MY "%Name%"