I'm sure this is a softball for those who are familiar with the Elastic Stack, but the docs I've read havent left it super clear.
I essentially am trying to push pcap files through the ELK stack to visualize packet information using Kibana.
I am not looking to monitor this real time, but rather have the following behavior:
I drop a pcap into a directory, and something picks it up (FileBeat? PacketBeat -I? LogStash?)
Since a pcap file isn't really useful, I might need to run it through tshark to produce readable json
I want this information in ElasticSearch
Use Kibana to make pretty graphs
From what I read PacketBeat allows for the -I option to take a pcap file as an input, but doesn't that only ship that single file? I want it to watch a directory as I drop pcaps. I guess what confused me is most of the docs talk about configuring an interface device to sniff in the packetbeat.yml
Anyway ideally I was thinking it would look something like this
packetbeat(watching for pcaps, spits out json) -> logstash (filters)-> elasticsearch (indexes)-> kibana (visualizes)
Is there a way to configure packetbeat to watch a dir for pcaps rather than an interface?
As of March 2021, you still can't do this natively with Packetbeat.
But you can easily "outsource" the watching of a directory tree to another tool, and have it call Packetbeat. Watchman (released by Facebook) is a good choice - it will keep track of files that have been processed. Then you could do something like the following to a) watch a directory and then b) take action when files are changed/added:
watchman watch /path/to/pcaps
watchman -- trigger ~/path/to/pcaps pcaptrigger '*.pcap' -- 'packetbeat -I'
Related
Is there possible to add a directory to Kodi using the command line? I've been looking for this with no luck so far.
What I'm looking for is to automate the process of adding a directory manually using command line. For some reason this doesn't appear to be a popular question out there; am I missing something?
Crawling the Kodi/XBMC forums and wiki show a few options... Here's what I've gathered...
Edit the Database Directly (not recommended)
Kodi stores this information in a sqlite database, however this location would be pretty tricky to manipulate yourself as it would require both knowledge of the path of each sqlite database file as well as the relationship of each column/table in each database file (assuming it's a strictly relational database file, which most are).
For example:
sqlite3 <path_to_kodi_preferences>/userdata/Database/MyVideos119.db
sqlite> .tables
actor movie_view studio_link
actor_link movielinktvshow tag
art musicvideo tag_link
bookmark musicvideo_view tvshow
country path tvshow_view
country_link rating tvshowcounts
director_link season_view tvshowlinkpath
episode seasons tvshowlinkpath_minview
episode_view sets uniqueid
files settings version
genre stacktimes writer_link
genre_link streamdetails
movie studio
Edit sources.xml
The official wiki mentions <path_to_kodi_preferences>/userdata/sources.xml for this but it still assumes you know how to manipulate an XML file programmatically and the community warns that this is potentially "invasive" and that the official addons/plugins aren't allowed to use this technique.
I dove into this and the XML seems like the way to go, for example, to add Videos:
<video>
<default pathversion="1"></default>
<source>
<name>Movies</name>
<path pathversion="1">/home/ubuntu/Movies/</path>
<allowsharing>true</allowsharing>
</source>
<source>
<name>Video Playlists</name>
<path pathversion="1">special://videoplaylists/</path>
<allowsharing>true</allowsharing>
</source>
+ <source>
+ <name>MyCustomDirectory</name>
+ <path pathversion="1">/home/ubuntu/MyCustomDirectory/</path>
+ <allowsharing>true</allowsharing>
+ </source>
</video>
... however comments suggest Kodi needs to be restarted and that this location still needs to be crawled/refreshed. There may be some "watchdog" add-ons that can do this for you.
Use the Add-On API
Another technique is to use the official Python API, such as through UpdateLibrary(database, path) however examples usually involve Python to call the API directly. Here's an example from the PlexKodiConnect GitHub project:
# Make sure Kodi knows we wiped the databases
xbmc.executebuiltin('UpdateLibrary(video)')
if utils.settings('enableMusic') == 'true':
xbmc.executebuiltin('UpdateLibrary(music)')
Since the simplest solution is often the best, I would recommend working on a way to automate the settings.xml file. Modifying XML files is well-documented in nearly all programming languages (to that point, you could technically brute-force and just inject an XML string at the given place without an xml parser 😈) and then handle the restart and refresh operations once the XML is confirmed as being updated properly.
Is there any way to execute a post project transfer event when transferring a project to a PLC?
I want to automatically change the value of a variable using fx the PVI interface every time I do a transfer.
I am not entirely sure what the usecase is for this. However the easiest way for some kind of post transfer script would be to utilize the Runtime Utility Center (RUC).
In the RUC, you can define instruction lists for a B&R PLC with an online connection. This includes instructions for transferring projects and setting values of process variables (PVs).
For transferring the project with RUC, you need to create a RUC package. This can be done under Settings/Export to Runtime Utility Center. You could also do this from the command line. More details in the help under Project management/Project installation/ Performing project installation/Export RUC Guid: cfe34190-f436-4c14-b06d-3a4ca39be7e7
This will create a zip, which you can then use in your RUC. For transfer command there is a wizard which activates when you double click on the command Transfer to target under Project installation The result is a line in the instruction list, which might look like this:
Transfer "C:\path\to\your\zip\project.zip", "InstallMode=Consistent InstallRestriction=AllowUpdatesWithoutDataLoss KeepPVValues=1 ExecuteInitExit=1"
After transferring you can write your PV. Under Process variable functions in the RUC you can find the command Write process variable. Also here there is a wizard and the result looks like this:
WriteVariable "taskname\VariableName", "USINT", "2"
I am using AS 4.4.6. There might be slight differences when using a other version.
I have a 3-axis system, which resembles the CNC-axis, and a PowerPanel from B&R. Now I plan to execute the g-code, which I read from a file, which is fe located in the Downloads folder. This will be done with fbs from B&R Automation Studio.
I got confused with the help, since it says in drive engineering/mapp Motion/technologie/function block/mpcnc3axis/description the part of moveProgram, where it redirects you to MC_BR_MoveProgram \ description. There stands that you can start the program with [Program name prefix] [File device name:] [Path] Program name syntax. Now I am not sure how to include this with my MpCnc3Axis code or if I can use the syntax also in MpCnc3AxisParType ProgramName part to start the program from fe Downloads Folder, since it didn't work with McPrgAdvParType InitProgramName.
Any links to related material would be appreciated.
I would also accept a solution with C, if it is easier.
Depending on if you are willing to buy software licences from B&R, running a CNC system is more or less easy.
The state of the art implementation is with mappMotion. It requires software licenses and gives you ready to go FUBs to execute nc files, jogging, direct movements, etc. This works the same way for different robot types and CNC kinematics. I recently uploaded a small implementation of a robot on GitLab: https://gitlab.com/kirni/bur_robotic_sample
The alternative is the old ARNC0 approach. No more new features are implemented, but it is free - as long as you do not need any functions that require a Dual Use agreement (i.e. 5 Axis CNC, etc.). The implementation is rather complex and is definitely to extensive to be answered completely at stackoverflow.
However Automation Studio ships with a sample for a two axis CNC. In the logical view you can add a new object Library Samples which opens a dialog where you can select Samples/Motion/CNC system (2 axis)/LibARNC0_Basic_ST.zip. This would be the best starting point, which is also documented in the help in MotionControl/ACP10 ARNC0/Examples/CNC or guid: 4620bb1c-8694-4f27-87d0-805686d7c0c8 as of AS 4.4.4.64.
I hope this helps!
This might be a question that may not be answered due to the nature of the external tool I am using (lack of documentation).
Basically, I am using a tool that pushes and pulls messages from the queue, more precisely - it pushes and pulls files. It worked perfectly for text files but when I tried pushing and then pulling a binary file - the pulled one was corrupted, it's size increased in comparsion with the original file (1.33 ratio).
For example moving a zip file wouldn't work...
I suppose it has something to do with the tools configuration, the only settings that can be changed regarding the problem are CCSID and encoding (UTF-8, Base16, etc.), I tried playing with both, unfortunately without success.
Tried using the following CCSIDs: 65535, 1208, 819
and encodings : UTF-8, Base16, Base64
In every case the binary file was corrupted after pulling it from the queue, I'm not entirely sure how the tool acomplishes that, it's written in Java, also I'm new to MQ so I tried searching for the correct options in IBM's docs but I haven't found anything that makes more sense than 65535 and Base16, yet it still doesn't work, could anyone with more experience with MQ tell if playing with these options makes sense at all in this case and if so - suggest what CCSID and encoding can I try to accomplish what Ive described above?
More information is really needed, but my suspicion is you are putting the message on the queue as a text message and playing around with encodings and ccsid's to try to get it right. You really need to know how the 'Java' app achieves this - is it using JMS (eg JMSBytesMessage) or base Java (something like setMessageData).
At a high level, there is a header on a message (The MD) which 'describes' the data - the MD format field. If you say the data is a string then MQ can convert between codepages should the getter request it etc. Put a tiny binary file into a message onto a queue, and browse the queue with amqsbcg or the GUI - what are the MD fields for format? What headers are on the payload - anything like RFH2's?
Put the same code in to give us a clue, or at least the amqsbcg output
We have several applications that use log4j for logging. I need to get a log4j parser working so we can combine multiple log files and run automated analysis on them. I'm not looking to reinvent the wheel, so can someone point me to a decent pre-existing parser? I do have the log4j conversion pattern if that helps.
If not, I'll have to roll our own.
I didn't realize that Log4J ships with an XML appender.
Solution was: specify an XML appender in the logging configuration file, include that output XML file as an entity into a well formed XML file, then parse the XML using your favorite technique.
The other methods had the following limitations:
Apache Chainsaw - not automated enough
jdbc - poor performance in a high performance distributed app
You can use OtrosLogViewer with batch processing. You have to:
Define you log format, you can use Log4j pattern layout parser or Log4j XmlLayout
Create java class that implements LogDataParsedListener. Method public void logDataParsed(LogData data, BatchProcessingContext context) will be called on every parsed log event.
Create jar
Run OtrosLogViewer with specifying your log processing jar, LogDataParsedListener implementation and log files.
What you are looking for is called SawMill, or something like it.
Log4j log files aren't really suitable for parsing, they're too complex and unstructured. There are third party tools that can do it, I believe (e.g. Sawmill).
If you need to perform automated, custom analysis of the logs, you should consider logging to a database, and analysing that. JDBC ships with the JdbcAppender which appends all messages to a database of your choice, but it has performance implications, and it's a bit flaky. There are other, similar, alternatives on the interweb, though (like this one).
You -can- use Log4j's Chainsaw V2 to process the various log files and collect them into one table, and either output those events as xml or use Chainsaw's built-in expression-based filtering, searching & colorizing support to slice & dice the logs.
Steps:
- Start Chainsaw V2
- Create a chainsaw configuration file by copying the example configuration file available from the Welcome tab - define one LogFilePatternReceiver 'plugin' entry for each log file that you want to process
- Start Chainsaw with that configuration
- Each log file will end up as a separate tab in the UI
- Pause the chainsaw-log tab and clear the events from that tab
- Create a new tab which aggregates the events from the various tabs by going to the 'view, crate custom expression logpanel' menu item and enter 'level >= DEBUG' in the box. It will create a new tab containing events from all of the tabs with level >= debug (which is why you cleared the chainsaw-log tab).
You can get an overview of the expression syntax used to filter, colorize and search from the tutorial (available from the Help menu).
If you don't want to use Chainsaw, you can do something similar - start a simple app that doesn't log but loads a log4j.xml config file with the 'plugin' entries you defined for the Chainsaw configuration, but also define a FileAppender with an xmllayout - all of the events received by the 'receivers' will be sent to the single appender.