I have two OPC clients subbed to about 160 items(temp sensors,pressure sensors,...) on a Siemens PLC(S7 1500) OPC UA server. Each client is on a different machine (one is on-site, other is hosted on cloud).
When we compare results on each client we can see small differences with our data. sometimes there is more data on one computer, sometimes there is more on the other. sometimes values are different for the same timestamp
Here is an example of a compare for 1 item between two clients
compare of values between two clients
We use the SourceTimestamp of the monitored item and its value coming from the OPC server to put it in the database.
This is not a big issue but I would like to understand why there is such a difference between two identical clients.
You've lost a lot of precision using the date/time format in your screenshots. Perhaps it's just a rounding issue and the underlying timestamps aren't exactly the same like you think they are?
Related
I'm using CCXT for some API REST calls for information and websockets. It's OK for 1 user, if I wanted to have many users using the platform, How would I go about an inhouse solution?
Currently each chart is either using websockets or rest calls, if I have 20 charts then thats 20 calls, if I increase users, then thats 20x whatever users. If I get a complete coin list with realtime prices from 1 exchange, then that just slows everything down.
Some ideas I have thought about so far are:
Use proxies with REST/Websockets
Use timescale DB to store the data and serve that OR
Use caching on the server, and serve that to the users
Would this be a solution? There's got to be a way to over come rate limiting & reducing the amount of calls to the exchanges.
Probably, it's good to think about having separated layers to:
receive market data (a single connection that broadcast data to OHLC processors)
process OHLC histograms (subscribe to internal market data)
serve histogram data (subscribe to processed data)
The market data stream is huge, and if you think about these layers independently, it will make it easy to scale and even decouple the components later if necessary.
With timescale, you can build materialized views that will easily access and retrieve the information. Every materialized view can set a continuous aggregate policy based on the interval of the histograms.
Fetching all data all the time for all the users is not a good idea.
Pagination can help bring the visible histograms first and limit the query results to avoid heavy IO in the server with big chunks of memory.
Nearly all PowerShell code leans heavily on a list of computer names. Maintaining this list is crucial for an up to date data center.
I used to gather computer names from my WSUS boxes. We have now switched to an inferior product which does not give up secrets easily! How are others gathering accurate lists of computer names with which to fuel your code?
Grabbing names from Active Directory works well, however not ALL servers are in the domain. However all servers need updates so let's get a list from the update server. Ahhh, there is the rub, inferior updating software! How is everyone gathering computer names for future processing?
For our datacenters, I always query each of the domain controllers to get a list of systems instead of relying on csv files (we no longer keep anything unjoined).
we used to have stand alone boxes, when we did I would simply get their IPs or netbios names by using the VMWare Power CLI to query that info from vCenter.
Alternatively, if you only have VMs, you can just go that route directly.
If you have stand-alone physical machines, then you would want to scan all of your IP range (after converting items you got from the previous two steps into IPs, so that you can exclude all IPs that are already accounted for by replies to netbios (to check the OS reported to make sure they are Windows systems)).
Alternatively, if all of the servers are in DNS, then just query the DNS entries from your domain controller, slap a parallel computer-exists check on the results, select unique where result is good and that is your list.
I'm not sure where you're getting this idea that people are composing CSV lists of machines for data centers. Most data centers I'm aware of will have virtualization stacks and have complex monitoring software such as the various flavors of SolarWinds products. Usually these will allow you to export a customized CSV of the machines they monitor. Of course, like anything else, there are quite a few of these monitoring services out there.
For small and mid-sized companies, admins may or may not maintain an inventory of company assets. I've been contracted to come out to companies who have let their asset management system slip and have no idea where everything is, and I usually use a combination of command line tools such as nmap, fping, and the PSADUC module to do some network discovery over several site days.
PowerShell isn't a great tool for data center use anyway, since most infrastructure uses some sort of Linux to host their virtual machines. Increasingly, the industry has moved towards containerization and microservices as well, which make maintaining boxes less and less relevant.
I just need a bit more clarity around tableau extract VS live. I have 40 people who will use tableau and a bunch of custom SQL scripts. If we go down the extract path will the custom SQL queries only run once and all instances of tableau will use a single result set or will each instance of tableau run the custom SQL separately and only cache those results locally?
There are some aspects of your configuration that aren't completely clear from your question. Tableau extracts are a useful tool - they essentially are temporary, but persistent, cache of query results. They act similar to a materialized view in many respects.
You will usually want to employ your extract in a central location, often on Tableau Server, so that it is shared by many users. That's typical. With some work, you can make each individual Tableau Desktop user have a copy of the extract (say by distributing packaged workbooks). That makes sense in some environments, say with remote disconnected users, but is not the norm. That use case is similar to sending out data marts to analysts each month with information drawn from a central warehouse.
So the answer to your question is that Tableau provides features that you can can employ as you choose to best serve your particular use case -- either replicated or shared extracts. The trick is then just to learn how extracts work and employ them as desired.
The easiest way to have a shared extract, is to publish it to Tableau Server, either embedded in a workbook or separately as a data source (which is then referenced by workbooks). The easiest way to replicate extracts is to export your workbook as a packaged workbook, after first making an extract.
A Tableau data source is the meta data that references an original source, e.g. CSV, database, etc. A Tableau data source can optionally include an extract that shadows the original source. You can refresh or append to the extract to see new data. If published to Tableau Server, you can have the refreshes happen on schedule.
Storing the extract centrally on Tableau Server is beneficial, especially for data that changes relatively infrequently. You can capture the query results, offload work from the database, reduce network traffic and speed your visualizations.
You can further improve performance by filtering (and even aggregating) extracts to have only the data needed to display your viz. Very useful for large data sources like web server logs to do the aggregation once at extract creation time. Extracts can also just capture the results of long running SQL queries instead of repeating them at visualization time.
If you do make aggregated extracts, just be careful that any further aggregation you do in the visualization makes sense. SUMS of SUMS and MINS of MINs are well defined. Averages of Averages etc are not always meaningful.
If you use the extract, than if will behave like a materialized SQL table, thus anything before the Tableau extract will not influence the result, until being refreshed.
The extract is used when the data need to be processed very fast. In this case, the copy of the source of data is stored in the Tableau memory engine, so the query execution is very fast compared to the live. The only problem with this method is that the data won't automatically update when the source data is updated.
The live is used when handling real-time data. Here each query is accessed from the source data, so the performance won't be as good as the extract.
If you need to work on a static database use extract else the live.
I am feeling from your question that you are worrying about performance issues, which is why you are wondering if your users should use tableau extract or use live connection.
From my opinion for both cases (live vs extract) it all depends on your infrastructure and the size of the table. It makes no sense to make an extract of a huge table that would take hours to download (for example 1 billion rows and 400 columns).
In the case all your users are directly connected on a database (not a tableau server), you may run on different issues. If the tables they are connecting to, are relatively small and your database processes well multiple users that may be OK. But if your database has to run many resource-intensive queries in parallel, on big tables, on a database that is not optimized for many users to access at the same time and located in a different time zone with high latency, that will be a nightmare for you to find a solution. On the worse case scenario you may have to change your data structure and update your infrastructure to allow 40 users to access the data simultaneously.
I need to make a simple simulation in Matlab. It consists of me (the client) sending a binary vector b to a server. This server contains already a vector x. I want it to calculate the innerproduct between vectors b and x and send it back to me.
Is it possible to create different independent servers in matlab (in one computer) that can exchange information with each other? is Using TCP/IP Server Sockets a good idea?
Please help me
Using TCP or UDP would make the simulation generalizable to servers on other computers. Interfaces for these protocols can be found in the Instrument Control Toolbox, but if you don't have that then several toolboxes on the File Exchange also provide interfaces. This one seems the most up to date.
An alternative is to communicate via a memory-mapped file, which MATLAB supports natively, but this would not work if the server were running on a different computer.
If you don't want to use any other Matlab toolboxes use could write a simple Java TCP sever and client and instantiate them from Matlab.
This previous question is on calling Java from Matlab.
There are many tutorials for setting up Java TCP servers and clients.
I was wondering what sort of benefits a nosql option might have either in lieu of, or in conjunction with a rdbms for a typical MMORPG. In particular, I'm curious as to how data integrity is preserved under nosql.
I'll give a few examples where I'm kind of confused as to how server logic would operate:
Scenario 1: Let's say player A and B attack player C at the same time. A player_stats.kill_count is updated for whichever player kills player C. Normally, one might handle this AttackPlayerRequest in the following way:
Begin Transaction
{
attacker.health = newAttackerHealth
defender.health = newDefenderHealth
if defender.Health == 0
{
attacker.stats.kills += 1
}
}
Scenario 2: Player A sells an item to an NPC vendor, and quickly attempts to drop the same item from their inventory to the ground. (Even if you're UI disallows this, they can certainly just put the events on the wire).
This list obviously goes on and effects just about every player-player player-world interaction. One last piece of information is that the server is threaded as we don't want any particularly lengthy request to block other players from being able to do simple things in the game.
I guess my big question here is, am I misunderstanding something about nosql wherein this a trivial problem? If not, is this beyond the domain of what nosql options are meant to solve? Under a typical MMO, where might one inject nosql options to increase server performance?
This all depends on how your game operates.
NoSQL
Farmville on facebook uses NoSQL. Farmville acts like 1 enormous server that everyone is on. Let's say Farmville uses 3 computers in its database cluster. Computer 1 in San Diego. Computer 2 in New York. Computer 3 in Moscow. I login to Facebook from San Diego so I
connect to the database here. I level up and log out. Eventually Computer 1 Will tell computer 2 and 3 about how I leveled but it could be up to an hour before someone in Russia would see those changes to my account.
NoSQL scaling is as simple as adding another computer to the cluster and telling the website in that region to use that computer.
SQL
There are several methods of making SQL Work but I will explain a way to make it similar to NoSQL
Each Data-center will have 10 game servers, each with its own SQL Database + 1 Sub Master Database. The Sub Master Database shared the information between all 10 servers so if you login to server 1 and make character John Doe, then logout, Server 10 will have your character if you login to that server next.
Sub Master then shares its information with the Master Server at your HQ. The Master Server will then share John Doe to every other sub master at all the other data-centers, and those sub masters will update their game servers. This configuration would allow for you to login to server Weed in San Francisco, play your character, logout, login on server Vodka in Moscow, and still see your character.
Games like World of Warcraft use SQL but only certain parts of the database are shared. This allows for reduced Database size on each computer and also reducing hardware requirements.
In a real life situation each Sub Master would have a backup Sub Master, and your Master server would have a few backup servers in the event that one server goes down your entire network would not lock up.
I think MMO servers would do a lot of the stuff you've listed in memory. I don't think they flush all of those out into the database. I think more hard-set events like receiving new gear or changing your gear layout would be saved.
It's a lot of data, but not nearly as much as every single combat interaction.
MMORPGs that are worried about latency typically run on a single server, and store everything except object model information in RAM. Anything with battle environments are worried about latency.
Anything that uses HDD, be it an RDBMS or NoSQL, is a bad idea in battle environments. Updating and sharing object customization, position, movement vector, orientation vector, health, power, etc... data between 10,000 users on a single server via any mechanism that uses HHD is just silly if you are worried about latency. Even if you have 2 people fighting in a simple arena, latency is still an issue and you should run on a single server and keep everything in memory. Everything that will be rendered should be on your PC or in the memory of your PC as well.
MMORPGs that are not worried about latency can run on multiple servers and any database you wish (NoSQL, MySQL, etc...)
If you're making a game like farmville or neopets, then yes, NoSQL or MySQL become valid options.
To my understanding, nosql technology is something you would use in a batch job context, and not in a real-time context. As a matter of fact, the open source H-base implementation sits on top of the hadoop distributed file system (hdfs), which is primarily used for read-only situations.
The idea is that you 'seek' a record by iterating over the whole data set you are keeping 'on disk' (on a distributed file system actually). For petascale datasets it is infeasible to keep things in memory, so you utilize massive amounts of disk reads to be able to 'seek' the data at all. The distributed nature makes sure (or facilitates) that this occurs in a timely fashion.
The principle is that you bring the processing to the distributed data, instead of pulling data from a very large database, sending it over the network and then process it on a local node. Pig (http://pig.apache.org/) and Hive (http://hive.apache.org/) are interfaces on top of hdfs, which allow for SQL-like queries, but in the background, they use mapreduce jobs. Interaction is slow, but this is not the point. The point is that the immense data set can be processed at all.
For gaming, this is obviously not the way to go. So you would expect that data to be fetched from the distributed file system, cached on a server, and used during a session, When the game has finished, all data would be committed back to the data set. (At least, that's how I imagine it would go).
Hdfs data sets are mostly processed at a certain point in time, after which the results are being published in one batch. For example, in a social network site, you would compile all connection data on a daily basis, search for all new connections between people, and when the operation has finished for all people in the data set, the results would be published in the typical 'X is now conected to Y' messages. This does not happen in real-time.