I just need a bit more clarity around tableau extract VS live. I have 40 people who will use tableau and a bunch of custom SQL scripts. If we go down the extract path will the custom SQL queries only run once and all instances of tableau will use a single result set or will each instance of tableau run the custom SQL separately and only cache those results locally?
There are some aspects of your configuration that aren't completely clear from your question. Tableau extracts are a useful tool - they essentially are temporary, but persistent, cache of query results. They act similar to a materialized view in many respects.
You will usually want to employ your extract in a central location, often on Tableau Server, so that it is shared by many users. That's typical. With some work, you can make each individual Tableau Desktop user have a copy of the extract (say by distributing packaged workbooks). That makes sense in some environments, say with remote disconnected users, but is not the norm. That use case is similar to sending out data marts to analysts each month with information drawn from a central warehouse.
So the answer to your question is that Tableau provides features that you can can employ as you choose to best serve your particular use case -- either replicated or shared extracts. The trick is then just to learn how extracts work and employ them as desired.
The easiest way to have a shared extract, is to publish it to Tableau Server, either embedded in a workbook or separately as a data source (which is then referenced by workbooks). The easiest way to replicate extracts is to export your workbook as a packaged workbook, after first making an extract.
A Tableau data source is the meta data that references an original source, e.g. CSV, database, etc. A Tableau data source can optionally include an extract that shadows the original source. You can refresh or append to the extract to see new data. If published to Tableau Server, you can have the refreshes happen on schedule.
Storing the extract centrally on Tableau Server is beneficial, especially for data that changes relatively infrequently. You can capture the query results, offload work from the database, reduce network traffic and speed your visualizations.
You can further improve performance by filtering (and even aggregating) extracts to have only the data needed to display your viz. Very useful for large data sources like web server logs to do the aggregation once at extract creation time. Extracts can also just capture the results of long running SQL queries instead of repeating them at visualization time.
If you do make aggregated extracts, just be careful that any further aggregation you do in the visualization makes sense. SUMS of SUMS and MINS of MINs are well defined. Averages of Averages etc are not always meaningful.
If you use the extract, than if will behave like a materialized SQL table, thus anything before the Tableau extract will not influence the result, until being refreshed.
The extract is used when the data need to be processed very fast. In this case, the copy of the source of data is stored in the Tableau memory engine, so the query execution is very fast compared to the live. The only problem with this method is that the data won't automatically update when the source data is updated.
The live is used when handling real-time data. Here each query is accessed from the source data, so the performance won't be as good as the extract.
If you need to work on a static database use extract else the live.
I am feeling from your question that you are worrying about performance issues, which is why you are wondering if your users should use tableau extract or use live connection.
From my opinion for both cases (live vs extract) it all depends on your infrastructure and the size of the table. It makes no sense to make an extract of a huge table that would take hours to download (for example 1 billion rows and 400 columns).
In the case all your users are directly connected on a database (not a tableau server), you may run on different issues. If the tables they are connecting to, are relatively small and your database processes well multiple users that may be OK. But if your database has to run many resource-intensive queries in parallel, on big tables, on a database that is not optimized for many users to access at the same time and located in a different time zone with high latency, that will be a nightmare for you to find a solution. On the worse case scenario you may have to change your data structure and update your infrastructure to allow 40 users to access the data simultaneously.
I am looking at using GeoTools to read shapefiles. The tutorial for using it is straightforward showing how to set a filter to "Filter filter = Filter.INCLUDE;" to specify everything.
I want to split up the reading for performance purposes on very large shape files. In essence I want to split the reading of the info in the DBF file from the reading of the "THE_GEOM" data. We have a lot of our own filtering already built and it is easier to just use it and then retrieve the actual geometry as required.
How do I specify a filter to retrieve all the DBF info without the geometry?
How do I specify a filter to retrieve the geometry without the DBF info? This isn't as important since it probably won't impact performance so much but I would like to know.
Thanks.
By design the GeoTools' Shapefile Datastore goes to great lengths to keep the geometry and the attributes (the DBF stuff) together. So you are going to have to poke around in the internals to be able to do this. So you could use a DBFFileReader and a ShapefileReader to split the reading up.
I would consider porting your filters to GeoTools as it gives you the flexibility to switch data sources later when Shapefiles prove too small or too slow. It might be worth looking at the CQL and ECQL classes to help out in constructing them.
If you are really dealing with large Shapefiles (>2Gb) then going with a proper spatial database like PostGIS is almost certainly going to give better performance and GeoTools will access the data in exactly the same way with exactly the same filters.
I am working on a map-matching/trajectory matching project. What I am unsure after reading a number of research papers is what is the most efficient data structure to store the road network (described by a weighted directed graph) so as to facilitate real time searching (Fast). I am getting things like grids, MTrees, Quadtrees...do I need a database in the backend for these. I am working on MATLAB at the moment but can shift languages. What are the programming languages used in actual satellite navigators.
Help will be much appreciated
There are many such index structures like K-D Trees, R-Trees etc. that can be built on the data that you store on the database. You can store your data on SQL Server and use these indexes. You might also want to take a look at SQL Server Spatial tools that helps to perform Spatial queries on your data.
I have loaded a massive set of tiles into a postgres database for use in a tile server. These were all loaded into bytea columns in PNG format.
I now find out that the tile server code needs these to be in GEOTiff format.
The command:-
gdal_translate -of GTiff -expand rgb -co COMPRESS=DEFLATE -co ZLEVEL=6
Works perfectly.
However, a massive amount of data is loaded already on a remote server. So is it possible for me to do the conversion within the database instead of retrieving each file and using gdal_translate on them individually? I understand that gdal is integrated with postgis 2.0 through the raster support which is installed on my server.
If not, any suggestions as to how to do this efficiently.
Is it possible to do in the database with an appropriate procedural language? I suppose. Additionally it is worth noting up front that gdal's support for Postgis goes rather one way.
To be honest the approach is likely to be "retrieve individual record, convert, restore using external image processing like you are doing." You might get some transaction benefit but this is likely to be offset by locks.
If you are going to go this route, you may find pl/java to be the most helpful approach since you can load any image processing library Java supports and use that.
I am, however, not convinced that this will be better than retrieve/transform/load.
Lets say that I have gps coordinates for 1000 stores and a short text to each one. What would be the best way to store this information? SQL? One more thing to consider is how to load the information into the app. It don't seem to be a smart thing to load everything direct, the best way seems to be loading the stores in the specific area but how do I search for those stores? Is that easy to do in SQL? As you see I don't have so much experience of database programming.
storing in SQLite3 file would be best for the moment as you have lots of data and through a db, you can fetch the required data (via query) back on demand.
as for the getting stores of specific area - assuming you want to locate nearby stores within 10 KM of radius.
This will not be hard to fetch from db.
I am no server expert but can give you a guideline which will work:-
you will be having current lat/long, you will have lat/long in the table as well for each row.
may be a server guy could help you create you a query with formula where you can fit the distance calculation formula in 'where' clause's condition and you will get the records of nearby (10 KMs) places records.