I would like to use DKAN to consume a DCAT file containing a data catalog and display it on the UI but am not sure how to do this or where instructions to do this are documented.
Are you trying to consume DCAT to create a data visualization or consume DCAT metadata to import datasets?
Either way DKAN doesn't currently support that, however if we understand the use case it may be something we can add to the roadmap. There is an open issue for harvesting DCAT metadata as datasets into DKAN here: https://github.com/GetDKAN/dkan/issues/868 . If you can comment on that issue with your use case that would be very helpful. We create our roadmap based off of community feedback so would like to hear from you.
Related
I want to fetch complete metadata of the given dataset through API call. Can anyone please suggest how to fetch metadata
You actually already manipulate and interact with various forms of metadata inside your Transforms Python builds today, but in a way that is structured to be safe when reading and writing.
While not all forms of metadata are possible to access today, this generally is because of the desire to ensure product stability and good version controls of your builds.
That said, if there's a certain interaction with metadata you'd like to see in the product, I'd recommend reaching out to your support engineers with a feature request so they can understand your request more specifically and discuss with our product teams.
I made a json api with using this => https://www.django-rest-framework.org/tutorial/quickstart/
All the articles I read teach the creation and use of api within its own platform, what I need is what I produce on the web, use it to in other platforms. I made my api but no idea about how to import it in other platforms..
so how can I use my own api in my c# windows form application or my flutter project
Any link, guide etc.
First of all you should be clear about why you need an api. If you need to transfer data from one system to another, pick a way that you know you can operate on both sides.
JSON or XML are just ways of representing data, first think about what you need and how can you transport that data between systems...After that the implementation should be clear.
I need a solution preferably something inbuilt (rather than creating my own application) which would help management search through multiple/all topics in Kafka. We are using Confluent Platform. Basically user should be able to search a keyword in a UI and it should search current log of multiple/all Kafka topics and return the data. All the topics in our environment use json to communicate.
So this search would enable us to track flow for example, multiple microservices send data from one system to another system and this flow can be tracked via a correlation id which is present in all the jsons. So if someone searchers this correlation id he should be able to see the messages involved in the flow. This search would have more use cases later on.
We need a solution which would have minimal coding involved. We would prefer to use a UI like Kibana.
On basic reading I suspect below solutions but not really sure as I am new to Confluent (used open-source Apache Kafka earlier):
Sol 1: use ksqldb. (need more help on how to use it)
Sol 2: Stream all topics data using Kafka Connect to Elastic Search by using inbuilt plugin and use Kibana on top of Elastic.
Kindly help to find the best case alternative.
You could use Elastic, sure.
You could also use Splunk, though.
There is also the pdk tool offered by Pilosa that creates a distributed index over Kafka events. (no affiliation)
Another option would be distributed tracing using interceptors between clients, not "on all topics", which sounds like what you actually need
I've done quite a bit of searching, but haven't been able to find anything within this community that fits my problem.
I have a MongoDB collection that I would like to normalize and upload to Google Big Query. Unfortunately, I don't even know where to start with this project.
What would be the best approach to normalize the data? From there, what is recommended when it comes to loading that data to BQ?
I realize I'm not giving much detail here... but any help would be appreciated. Please let me know if I can provide any additional information.
If you're using python, easy way is to read collection chunky and use pandas' to_gbq method. Easy and quite fast to implement. But better to get more details.
Additionally to the answer provided by SirJ, you have multiple options to load data to BigQuery, including loading the data to Cloud Storage, local machine, Dataflow any more as mentioned here. Cloud Storage supports data in multiple formats such as CSV, JSON, Avro, Parquet and more. You also have various options to load data using Web UI, Command Line, API or using the Client Libraries which support C#, GO, Java, Node.JS, PHP, Python and Ruby.
I am trying to combined data from multiple sources like RDBMS, xml files, web services using Marklogic. For this as I see from MarkLogic documentation on Metadata Catalog (https://www.marklogic.com/solutions/metadata-catalog/), Data Virtualization (https://www.marklogic.com/solutions/data-virtualization/) and Data Unification it is very well possible. But I am not able to get hold of any documentation describing how exactly to go about it or which tools to use to achieve this.
Looking for some pointers.
As the second image in the data-virtualization link shows, you need to ingest all data into MarkLogic databases. MarkLogic can then be put in between to become the single entry point for end user applications that need access to that data.
The first link describes the capabilities of MarkLogic to hold all kinds of data. It partly does so by storing them as-is, partly by extracting text and metadata for searching, partly by conversion (if you needs go beyond what the original format allows).
MarkLogic provides the general purpose MarkLogic Content Pump (MLCP) tool for this purpose. It allows ingesting zipped or unzipped files, and applying transformations if necessary. If you need to retrieve your data from a different database, you might need a bit more work to get that out. http://developer.marklogic.com holds tutorials, blogs, and tools that should help you get going. Searching the MarkLogic Mailing List through http://marklogic.markmail.org/ can provide answers as well.
HTH!
Combining a lot of data is a very broad topic. Can you describe a couple types of data you'd like to integrate, and what services or queries you would like to build on that data?