Snorby not display alerts on main page - snort

Building a Snort / Barnyard2 / Snorby setup.
Having trouble with getting snorby to see events.
Snort and barnyard2 are both running at boot.
Here is my config relevant to the problem.
Snort:
output unified2: filename snort.u2, limit 128
Barnyard2:
config reference_file: /usr/local/snort/etc/reference.config
config classification_file: /usr/local/snort/etc/classification.config
config gen_file: /usr/local/snort/etc/gen-msg.map
config sid_file: /usr/local/snort/etc/sid-msg.map
config hostname:localhost
config interface: eth1
input unified2
output database: log, mysql, user=snort password=snorbypass dbname=snorby host=localhost
Snorby:
snorby: &snorby
adapter: mysql
username: snort
password: "snorbypass"
host: localhost
rc.local:
ifconfig eth1 up
/usr/local/snort/bin/snort -D -u snort -g snort \
-c /usr/local/snort/etc/snort.conf -i eth1
/usr/local/bin/barnyard2 -c /usr/local/snort/etc/barnyard2.conf \
-d /var/log/snort \
-f snort.u2 \
-w /var/log/snort/barnyard2.waldo \
-D
Current status:
Right now, when the system boots, I can see both the snort and barnyard2 processes running as issued in the rc.local.
When browsing to localhost in a browser, I can login to Snorby and change password, etc...
I can also see the sensor listed under sensors.
When looking at the workers, there is one running. I have also deleted this from from withing the web ui and recreated it without any issues.
When looking in the database for snorby, I can "SELECT * from signature" and see alot of signatures listed here.
Also, I can see the size of the most recent /var/log/snort/snort.u2.1398021580 being constantly updated.
My barnyard.waldo is also in this directory and I can see it with data and you can see that it is no longer a text file, but a binary. This can be re-created by deleting the file re-creating a new barnyard2.waldo text file and restarting barnyard2. By doing so, the file will be turned into a binary file with the size of 2056.
The file ownership is snort:snort and the file permissions on the directory /var/log/snort is 666.
Possible problem??::
The only thing I can see that is not functioning correctly is when I stop barnyard2 and start without -D to see the startup.
I receive a repeating error:
--== Initialization Complete ==--
Using waldo file '/var/log/snort/barnyard2.waldo':
spool directory = /var/log/snort
spool filebase = snort.u2
time_stamp = 1398023768
record_idx = 0
Opened spool file '/var/log/snort/snort.u2.1398023768'
WARNING: No function defined to read header.
WARNING: No function defined to read header.
Closing spool file '/var/log/snort/snort.u2.1398023768'. Read 0 records
Opened spool file '/var/log/snort/snort.u2.1398024174'
WARNING: No function defined to read header.
Waiting for new data
WARNING: No function defined to read header.
WARNING: No function defined to read header.
WARNING: No function defined to read header.
WARNING: No function defined to read header.
WARNING: No function defined to read header.
WARNING: No function defined to read header.
There is very little on this error when I looked through Google, but I believe that barnyard2 is having trouble reading the snort,u2 file. You can see here that it seems to load it okay, but that is about it. Regardless, when looking in the Snorby UI, there are 0 events on the listed sensor.
Any ideas would be greatly appreciated.

Once I ran PulledPork to down load snort rules, I received a new sid-msg.map file for the version of snort currently running. Repeating error resolved.

Related

SchemaSpy - Connection Failure

I want to visualise my data base and tried using SchemaSpy, but I can't connect to the data base. It's a mysql data base, running on my computer, accessible under localhost and I called SchemaSpy with the following command:
java -jar schemaspy-6.0.0.jar -t mysql -dp postgresql-42.2.7.jar -db <name> -host <host> -p <port> -s <name> -u <user> -p <password> -o ./output/
I downloaded postgresql-42.2.7.jar as a driver and set the respective parameter in the call. I also tried not setting a driver at all, but it didn't change the output.
I get the following error message after trying to run SchemaSpy:
SchemaSpy generates an HTML representation of a database schema's relationships.
SchemaSpy comes with ABSOLUTELY NO WARRANTY.
SchemaSpy is free software and can be redistributed under the conditions of LGPL version 3 or later.
http://www.gnu.org/licenses/
INFO - Starting Main v6.0.0 on ####### with PID 10774 (/path/to/file started by ##### in /path/to/file)
INFO - The following profiles are active: default
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (jar:file:/path/to/file/schemaspy-6.0.0.jar!/BOOT-INF/lib/spring-core-4.3.13.RELEASE.jar!/) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
INFO - Found configuration file: schemaspy.properties
INFO - Started Main in 1.397 seconds (JVM running for 1.944)
INFO - Starting schema analysis
WARN - Connection Failure
I also tried setting the --illegal-access=warn parameter, but no changes.
Does anyone know what problem might be?
Thanks a lot!

Enable FTRACE in ARM to trace realtime characteristic of system

I followed this article to enable FTRACE
https://lwn.net/Articles/365835/
to test a realtime system, my system uses arm cortexa15 (Description: https://mp.renesas.com/en-us/rzg/marketplace/board/RZGB000003.html)
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_STACK_TRACER=y
CONFIG_DYNAMIC_FTRACE=y
But, it didn't work, caused the system hang-up when starting kernel.
Even referred How to Enable or configure ftrace module
I would like to test latency in the realtime system with cyclictest (option -b to trigger FTRACER)
cyclictest -a -t -n -p99 -f -b100
It generated dump message:
INFO: debugfs mountpoint: /sys/kernel/debug/tracing/
WARN: tracing_enabled or tracing_on not found
debug fs not mounted, TRACERs not configured?
could not set ftrace_enabled to 0
FATAL: Can't open /sys/kernel/debug/tracing/available_tracers for reading
I repeated a next step to enable a group of tracer configs:
CONFIG_FTRACE=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FUNCTION_TRACER=y
CONFIG_IRQSOFF_TRACER=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_PREEMPT_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_ENABLE_DEFAULT_TRACERS=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_TRACER_SNAPSHOT=y
CONFIG_STACK_TRACER=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_TRACEPOINT_BENCHMARK=y
CONFIG_BACKTRACE_SELF_TEST=y
CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_LL=y
The result still was the same. Kernel hung and didn't show anything.
Anyone who deal with realtime system and Ftrace can help ? Thanks.
I resolve my problem. Below is a part of my defconfig file.
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_TRACEPOINTS=y
CONFIG_STACKTRACE=y
CONFIG_NOP_TRACER=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACE_CLOCK=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_GENERIC_TRACER=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_TRACER_SNAPSHOT=y
CONFIG_STACK_TRACER=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_FTRACE_MCOUNT_RECORD=y
After enabling Ftrace tool, the culprit can then be found in the trace output at /sys/kernel/debug/tracing/trace. The kernel function that was executed just before a latency of more than 100 microseconds was detected is marked with an exclamation mark.

Hawq init failed -- "postgres" is needed by initdb

After I build incubator-hawq on Centos7.1, I tried to init it. But the error below occurs:
20160516:18:10:43:002036 hawqinit.sh:host-172-16-0-105:hawqadmin-[INFO]:-Loading hawq_toolkit...
ALTER ROLE
20160516:18:10:44:001766 hawq_init:host-172-16-0-105:hawqadmin-[INFO]:-20160516:18:10:43:002036 hawqinit.sh:host-172-16-0-105:hawqadmin-[INFO]:-Loading hawq_toolkit...
20160516:18:10:44:001766 hawq_init:host-172-16-0-105:hawqadmin-[INFO]:-Master init successfully
20160516:18:10:44:001766 hawq_init:host-172-16-0-105:hawqadmin-[INFO]:-Init segments in list: ['hawq-master']
20160516:18:10:44:001766 hawq_init:host-172-16-0-105:hawqadmin-[DEBUG]:-Start to init segment on node 'hawq-master'
20160516:18:10:44:001766 hawq_init:host-172-16-0-105:hawqadmin-[INFO]:-Total segment number is: 1
fgets failure: Success
The program "postgres" is needed by initdb but was either not found in the same directory as "/usr/hawq/bin/initdb" or failed unexpectedly.
Check your installation; "postgres -V" may have more information.
20160516:18:10:45:002318 hawqinit.sh:host-172-16-0-105:hawqadmin-[ERROR]:-Postgres initdb failed
20160516:18:10:45:002318 hawqinit.sh:host-172-16-0-105:hawqadmin-[ERROR]:-Segment init failed on host-172-16-0-105
20160516:18:10:45:001766 hawq_init:host-172-16-0-105:hawqadmin-[INFO]:-20160516:18:10:45:002318 hawqinit.sh:host-172-16-0-105:hawqadmin-[ERROR]:-Postgres initdb failed
20160516:18:10:45:002318 hawqinit.sh:host-172-16-0-105:hawqadmin-[ERROR]:-Segment init failed on host-172-16-0-105
20160516:18:10:45:001766 hawq_init:host-172-16-0-105:hawqadmin-[ERROR]:-HAWQ init failed on hawq-master
20160516:18:10:46:001766 hawq_init:host-172-16-0-105:hawqadmin-[INFO]:-0 of 1 segments init successfully
20160516:18:10:46:001766 hawq_init:host-172-16-0-105:hawqadmin-[ERROR]:-Segments init failed, exit
When I type the command, the below shows:
[hawqadmin#host-172-16-0-105 hawqAdminLogs]$ postgres -V
postgres (HAWQ) 8.2.15
Any advice? Thanks!
If "postgres -V" works, that means the postgres binary is good.
Before you do "hawq init cluster", please make sure:
1) $GPHOME in greenplum_path.sh is correctly set to the directory of hawq binary, i.e, /usr/hawq in your case
2) source $GPHOME/greenplum_path.sh
3) check if initdb and postgres binary is in $GPHOME/bin
From the error you pasted above, 2 possible causes:
(1) The binary postgres called is not /usr/hawq/bin/postgres, You can use which postgres to check the path.
(2) The dependent lib for postgres may be wrong. You can use ldd for linux or otool for mac to print all dependent lib paths, and check them.
Moreover, if any error when init hawq, please check log in ~/hawqAdminLogs/, you may find out the specific error message.
Hope it will help you to find out the root cause.
Recently I faced same error while initializing cluster.
Postgres -V showed correct version, which postgres showed /usr/local/hawq/bin/postgres, also the path was already set, still faced above error.
Finally resolved by setting LD_LIBRARY_PATH to /usr/local/hawq/lib/ and sourced it via .bashrc file.
Looks like you might have installed hawq binaries in different directory . Please check the following
1.Make sure you have all the right PATH set
Check hawq initdb binaries are there in /usr/hawq/bin/ directory
make sure you have successed compile hawq and install them
check postgres is in the same dir with initdb
if there are more than 1 postgres in your pc, make sure the path of postgres(the same dir with initdb) is in your PATH.

FATAL org.apache.hadoop.conf.Configuration - error parsing conf file: org.xml.sax.SAXParseException

I'm trying to run pig locally, installed using homebrew, to test a script. However, I get the following error when I attempt to run a simple dump from the interactive prompt pig -x local:
2012-07-16 23:20:40,447 [Thread-7] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
[Fatal Error] :63:85: Character reference "&#2" is an invalid XML character.
2012-07-16 23:20:40,688 [Thread-7] FATAL org.apache.hadoop.conf.Configuration - error parsing conf file: org.xml.sax.SAXParseException: Character reference "&#2" is an invalid XML character.
The same load/dump works fine on Elastic MapReduce.
I can't find any XML config files, and I've tried with both version 0.9.2 and 0.10.0
What am I missing?
Edit: Just checked a direct download (vs. homebrew) and it doesn't seem to work either
You should check that your Hadoop configuration files have correct configuration data.
Have a look in your hadoop/conf directory.
Have a look inside:
hdfs-site.xml
mapred-site.xml
core-site.xml
Finally worked out what the problem was. I ended up having to use dtruss -p on the pig/java process. This revealed a temporary directory and dynamically generated xml files. Once the temporary directory was discovered, it all fell quickly into place.
It was picking up the proxy excludes from my network connections, which had, as far as I can tell, &#2 (http://www.fileformat.info/info/unicode/char/02/index.htm) embedded in it. How this invalid value came to be in my network preferences in the first place, I haven't the faintest clue.
The value was then being pulled into dynamically generated files, for example /tmp/hadoop-vertis/mapred/staging/vertis-1005847898/.staging/job_local_0001/job.xml.
The offending lines:
<property><name>ftp.nonProxyHosts</name><value>localhost|*.localhost|127.0.0.1|h|*.h</value></property>
<property><name>socksNonProxyHosts</name><value>localhost|*.localhost|127.0.0.1|h|*.h</value></property>
<property><name>http.nonProxyHosts</name><value>localhost|*.localhost|127.0.0.1|h|*.h</value></property>

Error while configuring Google App Engine on Pydev in Eclipse

i am trying to configure Google App Engine on Eclipse and use it to run a python application locally (on the local host):
for this i used following tutorial as a guide:
http://www.mkyong.com/google-app-engine/google-app-engine-python-hello-world-example-using-eclipse/
i followed the steps properly but when i try to use the configuration i get errors the console output is:
Console Output:
C:\Program Files (x86)\Google\google_appengine\google\appengine\api\search\search.py:232: UserWarning: DocumentOperationResult._code is deprecated. Use OperationResult._code instead.
'Use OperationResult.%s instead.' % (name, name))
C:\Program Files (x86)\Google\google_appengine\google\appengine\api\search\search.py:232: UserWarning: DocumentOperationResult._CODES is deprecated. Use OperationResult._CODES instead.
'Use OperationResult.%s instead.' % (name, name))
WARNING 2012-06-20 14:53:01,451 rdbms_mysqldb.py:74] The rdbms API is not available because the MySQLdb library could not be loaded.
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\dev_appserver.py", line 126, in
run_file(file, globals())
File "C:\Program Files (x86)\Google\google_appengine\dev_appserver.py", line 122, in run_file
execfile(script_path, globals_)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_main.py", line 694, in
sys.exit(main(sys.argv))
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_main.py", line 582, in main
root_path, {}, default_partition=default_partition)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 3142, in LoadAppConfig
raise AppConfigNotFoundError
google.appengine.tools.dev_appserver.AppConfigNotFoundError
The configuration i am using is:
Windows 7 64 bit
python 2.7
Eclipse Helios
What could be the possible mistake in configuring the GAE?
Additional info : when i try to use the project with GAE manually(ie by using the launcher) it works
Update:
i experimented and discovered that since the workstation and the python installation folder is not in the same directory i get these errors
got the hint from here:
File
"C:\Program Files(x86)\Google\google_appengine\google\appengine\tools\dev_appserver.py"
line 582, in main
root_path, {}, default_partition=default_partition)
but when i made another workspace in the same partition i got this as console output and the the local host is still not working
output
C:\Program Files (x86)\Google\google_appengine\google\appengine\api\search\search.py:232: UserWarning: DocumentOperationResult._code is deprecated. Use OperationResult._code instead.
'Use OperationResult.%s instead.' % (name, name))
C:\Program Files (x86)\Google\google_appengine\google\appengine\api\search\search.py:232: UserWarning: DocumentOperationResult._CODES is deprecated. Use OperationResult._CODES instead.
'Use OperationResult.%s instead.' % (name, name))
WARNING 2012-06-20 17:20:56,719 rdbms_mysqldb.py:74] The rdbms API is not available because the MySQLdb library could not be loaded.
Runs a development application server for an application.
dev_appserver.py [options]
Application root must be the path to the application to run in this server.
Must contain a valid app.yaml or app.yml file.
Options:
--address=ADDRESS, -a ADDRESS Address to which this server should bind(Defaultlocalhost).
--clear_datastore, -c Clear the Datastore on startup. (Default false)
--debug, -d Use debug logging. (Default false)
--help, -h View this helpful message.
--port=PORT, -p PORT Port for the server to run on. (Default 8080)
--allow_skipped_files Allow access to files matched by app.yaml's
skipped_files (default False)
--auth_domain Authorization domain that this app runs in.
(Default gmail.com)
--backends Run the dev_appserver with backends support
(multiprocess mode).
--blobstore_path=DIR Path to directory to use for storing Blobstore
file stub data.
--clear_prospective_search Clear the Prospective Search subscription index
(Default false).
--datastore_path=DS_FILE Path to file to use for storing Datastore file
stub data.
(Defaultc:\users\anukoo~1\appdata\local\temp\dev_appserver.datastore)
--debug_imports Enables debug logging for module imports, showing
search paths used for finding modules and any
errors encountered during the import process.
--default_partition Default partition to use in the APPLICATION_ID.
(Default dev)
--disable_static_caching Never allow the browser to cache static files.
(Default enable if expiration set in app.yaml)
--disable_task_running When supplied, tasks will not be automatically
run after submission and must be run manually
in the local admin console.
--enable_sendmail Enable sendmail when SMTP not configured.
(Default false)
--high_replication Use the high replication datastore consistency
model. (Default false).
--history_path=PATH Path to use for storing Datastore history.
(Default c:\users\anukoo~1\appdata\local\temp\dev_appserver.datastore.history)
--multiprocess_min_port When running in multiprocess mode, specifies the
lowest port value to use when choosing ports. If
set to 0, select random ports.
(Default 9000)
--mysql_host=HOSTNAME MySQL database host.
Used by the Cloud SQL (rdbms) stub.
(Default 'localhost')
--mysql_port=PORT MySQL port to connect to.
Used by the Cloud SQL (rdbms) stub.
(Default 3306)
--mysql_user=USER MySQL user to connect as.
Used by the Cloud SQL (rdbms) stub.
(Default )
--mysql_password=PASSWORD MySQL password to use.
Used by the Cloud SQL (rdbms) stub.
(Default '')
--mysql_socket=PATH MySQL Unix socket file path.
Used by the Cloud SQL (rdbms) stub.
(Default '')
--persist_logs Enables storage of all request and application
logs to enable later access. (Default false).
--require_indexes Disallows queries that require composite indexes
not defined in index.yaml.
--show_mail_body Log the body of emails in mail stub.
(Default false)
--skip_sdk_update_check Skip checking for SDK updates. If false, fall back
to opt_in setting specified in .appcfg_nag
(Default false)
--smtp_host=HOSTNAME SMTP host to send test mail to. Leaving this
unset will disable SMTP mail sending.
(Default '')
--smtp_port=PORT SMTP port to send test mail to.
(Default 25)
--smtp_user=USER SMTP user to connect as. Stub will only attempt
to login if this field is non-empty.
(Default '').
--smtp_password=PASSWORD Password for SMTP server.
(Default '')
--task_retry_seconds How long to wait in seconds before retrying a
task after it fails during execution.
(Default '30')
--use_sqlite Use the new, SQLite based datastore stub.
(Default false)
Invalid arguments
seems like the arguments to dev_appserver.py are incorrect any ideas
If you did the same mistake as mine,then its a very high chance that you have space in the name of your directory
The warnings about deprecated things in search can be ignored. As can the message about the rdbms API if you don't plan on using that.
AppConfigNotFoundError happens when there is no app.yaml in the directory passed to dev_appserver.py. If you followed those instructions, then your app.yaml would be in the 'src' directory, and the 'program arguments' in the build command would be ${project_loc}/src - is this the case? When you run from the command-line, and see it work, what command are you running, and from what location?