I would like to use a open source data cataloging tool in my company and was evaluating Amundsen for the same. I have a lot of sql server on my onPremise and would like to catalog all those in Amunsen. Currently for POC I am using docker containers for Amundsen on my local machine.
Could not find any help in cataloging my sql server tables. Can anyone please help me on how to do it in Amundsen, Lyft
ran into the same problem.
found that the SQL server extractor had a few bugs, so I made a pull request to fix them and added a sample script to build.
https://github.com/lyft/amundsendatabuilder/pull/272
Related
Currently we are using Oracle 19c external table functionality on-prem whereby CSV files are loaded to a specific location on DB server and they get automatically loaded into an oracle external table. The file location is mentioned as part of the table DDL.
We have a requirement to migrate to azure managed postgresql. As per checking the postgresql documentation, similar functionality as oracle external table can be achieved in standalone postgresql using "foreign tables" with the help of file_fdw extension. But in azure managed postgresql, we cannot use this since we do not have access to the DB file system.
One option I came across was to use azure data factory but that looks like an expensive option. Expected volume is about ~ 1 million record inserts per day.
Could anyone advise possible alternatives? One option I was thinking was to have a scheduled shell script running on an azure VM which loads the files to postgresql using PSQL commands like \copy. Would that be a good option for the volume to be supported?
Regards
Jacob
We have one last option that could be simple to implement in migration. We need to use Enterprise DB (EDB) which will avoid the vendor lock-in and also it is free of cost.
Check the below video link for the migration procedure steps.
https://www.youtube.com/watch?v=V_AQs8Qelfc
I am trying to follow quickstart to setup SQL Server (not LocalDb version of SQLServer that comes with Visual Studio) as my data store. Looks like that two databases will be needed - one for configuration and the other for operation. But my problem is that I couldn't figure out what db names I should use. I created two databases using names I came up with and ran the scripts I downloaded from quickstart to create all the tables. Now, when I try to make connection, I think I will need to specify db names in my connection string, don't I? What should I use to replace the original connection string provide by quickstart - "Data Source=(LocalDb)\MSSQLLocalDB;database=IdentityServer4.Quickstart.EntityFramework-4.0.0;trusted_connection=yes;" ?
You can have one database for both. But in general I would keep the configuration part in memory if the number of clients is small. Why spend hours keeping the config in a database for just a few clients and resources?
Better to just keep the users and persisted grants in a database.
I am new in the form and also new in postgresql.
Normally I use MySQL for my project but I’ve decided to start migrating towards postgresql for some valid reasons which I found in this database.
Expanding on the problem:
I need to analyze data via some mathematical formulas but in order to do this I need to get the data from the software via the API.
The software, the API and Postgresql v. 11.4 which I installed on a desktop are running on windows. So far I’ve managed to take the data via the API and import it into Postgreql.
My problem is how to transfer this data from
the local Postgresql (on the PC ) to a web Postgresql (installed in a Web server ) which is running Linux.
For example if I take the data every five minutes from software via API and put it in local db postgresql, how can I transfer this data (automatically if possible) to the db in the web server running Linux? I rejected a data dump because importing the whole db every time is not viable.
What I would like is to import only the five-minute data which gradually adds to the previous data.
I also rejected the idea of making a master - slave architecture
because not knowing the total amount of data, on the web server I have almost 2 Tb of hard disk while on the local pc I have only one hard disk that serves only to take the data and then to send it to the web server for the analysis.
Could someone please help by giving some good advice regarding how to achieve this objective?
Thanks to all for any answers.
So I've started a SQL Server database project inside VS 2012. I have done this for other databases already but not related to Service Broker.
For testing I had already created db, queues, etc through a T-SQL script including Message Types which was in an XML format. i.e.
[//blah.com/Items/RequestItem]
When I try to do something like this in the DB Project it's not allowing me too due to special chars.
Anyone done this? Gotten around it?
Is there a way to simply put my already created T-SQL file in the database project and have it use it?
See my comment above. I was able to import the script by Right clicking on the database Project.
I have sql azure database and to create and edit database using portal is very boring task due to it' user interface, when i will connect it with my local sql server R2 then i can not able to edit , create table from there.
Is there any way to make it possible , Please give me some solution for that
At this time, the two options available are the web user interface (which will be improved over time) and SQL Server Mgmt Studio (using queries; no user interface) for which SQL Azure support will also improve over time.
After all i found one 3rd party client to manage SQL Azure and that is RazorSQL- Awesome tool! I have write down about it in my blog, see here
Navicat is a commercial application that offers access.
http://www.navicat.com/products/navicat-for-sqlserver
Personally I vastly prefer it to the Microsoft web interface