(I'm afraid I'm probably about to reveal myself as completely unfit for the task at hand!)
I'm trying to setup a Redshift cluster and database to help manage data for a class/group project.
I have a dc2.large cluster running with either default options, or what looked like the most generic in the couple of place I was forced to make entries.
I have downloaded Aginity (Win64) as it is described as being specialized for Redshift. That said, I can't find any instructions for connecting using it. The connection dialog requests the follwoing:
Server: using the endpoint for my cluster (less :57xx at the end).
UserID: the Master username for the database defined for the cluster.
Password: to match the UserID
SSL Mode (Disable, Allow, Prefer, Require): trying various options
Database: as named in cluster setup
Port: as defined in cluster setup
I can't get it to connect ("failed to establish connection") and don't know if I'm entering something wrong in Aginity or if I haven't set up my cluster properly.
Message: Failed to establish a connection to 'abc1234-smtm.crone7m2jcwv.us-east-1.redshift.amazonaws.com'.
Type : Npgsql.NpgsqlException
Source : Npgsql
Trace : at Npgsql.NpgsqlClosedState.Open(NpgsqlConnector context, Int32 timeout)
at Npgsql.NpgsqlConnector.Open()
at Npgsql.NpgsqlConnection.Open()
at Aginity.MPP.Common.BaseDataProvider.get_Connection()
at Aginity.MPP.Common.BaseDataProvider.CreateCommand(String commandText, CommandType commandType, IDataParameter[] commandParams)
at Aginity.MPP.Common.BaseDataProvider.ExecuteReader(String commandText, CommandType commandType, IDataParameter[] commandParams)
--- Inner Exception: ---
......
It seems there is not enough information going into Aginity to authorize connection to my cluster - no account credential are supplied. For UserID, am I meant to enter the ID of a valid user? Can I use the root account? What would the ID look like? I have setup a User with FullAccess to S3 and Redshift, then entered the UserID in this format
arn:aws:iam::600123456789:user/john
along with the matching password, but that hasn't worked either.
The only training/tutorial I have been able to find/do on this is the Intro AWS direct you to, at https://qwiklabs.com/focuses/2366, which uses a web-based client that I can't find outside of the tutorial (pgweb).
Any advice what I am doing wrong, and how to do it right?
Well, I think I got it working - I haven't had a chance to see if I can actually create table yet, but it seems to be connected. I had to allow inbound traffic from outside the VPC, as per the above snapshot.
I'm guessing there's a better way than opening it up to all IP addresses, but I don't know the users' (fellow team members) IPs, and aren't they all subject to change depending on the device they're using to connect?
How does one go about getting inside the VPC to connect that way, presumably more securely?
Related
So I've been trying to publish my first project to azure. I've got everything set-up, a service app and a sql database.
My initial page loads properly(It's the standard view for a .net core web application).
The first thing I need to do is register a new user. Whenever I try through my azure app (myapp.azurewebsites.net) it fails and the logs says it's db related.
However I try the same thing by running the application on my machine in production environment, again connected to the azure sql server and everything works perfectly. I can register users, I can create posts, I can edit them. The allow access to azure services option is turned on. This error is from the eventlogs. I have not included the stacktrace.
Category: Microsoft.EntityFrameworkCore.Query EventId: 10100 RequestId: 800001be-0000-ba00-b63f-
84710c7967bb RequestPath: /Identity/Account/Register SpanId: |1e5a93ae-43f424904f38ea9f. TraceId:
1e5a93ae-43f424904f38ea9f ParentId: ActionId: c3430236-e61c-4785-a3c3-4f60ba115b6e ActionName:
/Account/Register An exception occurred while iterating over the results of a query for context type
'MyApp.Data.ApplicationDbContext'. Microsoft.Data.SqlClient.SqlException (0x80131904): Server
name cannot be determined. It must appear as the first segment of the server's dns name
(servername.database.windows.net). Some libraries do not send the server name, in which case the
server name must be included as part of the user name (username#servername). In addition, if both
formats are used, the server names must match.
Those are the different ways I tried to add the connection string to the appsettings.json file. (Server name, catalog, user and password have been replaced, they are written correctly in the appsettings file)
Server=tcp:servername.database.windows.net,1433;Initial Catalog=db;Persist Security Info=False;
User ID=user#server;Password=mypassword;
MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;
Server=tcp:servername.database.windows.net,1433;Initial Catalog=db;Persist Security Info=False;
User ID=user;Password=mypassword;
MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;
Data Source=tcp:server.database.windows.net,1433;
Initial Catalog=db;User Id=#server.database.windows.net;Password=password;
Alright so after a day and a half, I finally managed to fix it. The solution is rather simple and it is most likely my newbie mistake, that caused so much trouble.
I was following a tutorial for setting up the application and database connection after that. In the tutorial, the connection string that was being used, was the default one, found in the "myApp -> Configuration -> Connection strings", the format was:
Data Source=tcp:server.database.windows.net,1433;
Initial Catalog=db;User Id=#server.database.windows.net;Password=password;
This one was working in the guide, but not for me. So what I did, was go to my "sqldb -> connection strings" and copied the one provided there. I then went back to the app configuration and added it as a new configuration string using SqlServer as the Type.
This string was in the format:
Server=tcp:servername.database.windows.net,1433;Initial Catalog=db;Persist Security Info=False;
User ID=user;Password=mypassword;
MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;
After that, the app started working properly.
In the PostgreSQL documentation https://www.postgresql.org/docs/10/libpq-connect.html, it has been said that multiple hosts can be specified in a single connection string such that all the hosts will be tried in order one after the other until one of the server gets succeeds.
But when i tried to implement the same setting in the tag present in my ASP.net web.config file, it is throwing error as no such host name. I am using NpgSQL provider in order to connect to PostgreSQL database.
I need to add multiple server names in the connection string such that if the server#1 fails then it should try for the next server server#2 immediately provided in the order until it succeeds
Can you please suggest on how multiple hosts can be provided in the connection string?
The Npgsql driver does not currently support this functionality. The issue tracking this is https://github.com/npgsql/npgsql/issues/732, I'm still hoping we can get this into the next release but there's a lot going on.
Load balancing and failover is avaialble in Npgsql version 6. At the time of writing v.6 is in preview.
Simple failover example (server2 is only used if a connection could not be established to server1):
Host=server1,server2;Username=test;Password=test
Example with load balancing (round robin I guess):
Host=server1,server2,server3,server4,server5;Username=test;Password=test;Load
Balance Hosts=true;Target Session Attributes=prefer-standby
https://www.npgsql.org/doc/failover-and-load-balancing.html
I have a query I'd like to run regularly in Redshift. I've set up an AWS Data Pipeline for it.
My problem is that I cannot figure out how to access Redshift. I keep getting "Unable to establish connection" errors. I have an Ec2Resource and I've tried including a subnet from our cluster's VPC and using the Security Group Id that Redshift uses, while also adding that sg-id to the inbound part of the rules. No luck.
Does anyone have a from-scratch way to set up a data pipeline to run against Redshift?
How I currently have my pipeline set up
RedshiftDatabase
Connection String: jdbc:redshift://[host]:[port]/[database]
Username, Password
Ec2Resource
Resource Role: DataPipelineDefaultResourceRole
Role: DataPipelineDefaultRole
Terminate after: 20 minutes
SqlActivity
Database: [database] (from Connection String)
Runs on: Ec2Resource
Script: SQL query
Error message
Unable to establish connection to jdbc:postgresql://[host]:[port]/[database] Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Ok, so the answer lies in Security Groups. I had to find the Security Group my Redshift cluster is in, and then add that as a value to "Security Group" parameter on the Ec2Resource in the DataPipeline.
Ec2Resource
Resource Role: DataPipelineDefaultResourceRole
Role: DataPipelineDefaultRole
Terminate after: 20 minutes
Security Group: sg-XXXXX [pull from Redshift]
Try opening inbound rules to all sources, just to narrow down possible causes. You've probably done this, but make sure you've set up your jdbc driver and configurations according to this.
I did move the databases from our ServiceBus test enviroment.
I started by leaving the farm with the single node, then I moved the databases.
After rejoining the farm I see that GatewayDBConnectionString is till pointing to the old one.
I can't find any valid PowerShell command to reconfigure the value in question.
Anyone know how to fix this?
Thank you in advance.
To answer this I will need you to understand this a bit more - and hence giving a high-level overview of Service Bus 1.1 Server farm configuration:
Service Bus Server 1.1 is a platform where users can create highly-durable distributed Pub-Sub (messaging Queues/Topics) entities. In simple words - the main job of this is to translate the Compute (your VMs) and Data (your MsgContainer databases) into messaging functionality Durable Queues and Topics. So, in short - the configuration wizard or the Powershell cmdlets used to configure ServiceBus 1.1 Server will try to take the VMs and Databases from you.
The Db SBManagementDB is considered to be the authoritative source of truth for any Farm level configuration -> like Nodes that are part of the Farm (Store.Nodes), Ports opened on each of the nodes, Gateway database connection string (Cluster Config) etc. Also pl. note that - as per the Windows Server product guidelines - any information that has to be securely persisted will be encrypted - so as the Gateway DB connection String.
a) when you did New-SBFarm (with a Gateway DB connection string) - you have essentially communicated to SBMgmtDB - the Gateway DB Server, database name etc.
b) when you do Add-SBHost - again you have communicated to SBMgmtDb that you want to add one Node to this Farm
Gateway db connection string is the one place for Truth for all Gateway Services to find any run-time info -> like Container Databases, entity to container mapping etc.
again, when you do New-SBMessageContainer PSCmdlet --> you communicated to SBGatewayDB that you are adding one db
Now, with this background - lets see how the action you did above will take into effect:
- When you moved all the Databases to a different Server - you changed the Gateway Database connection string - But the Gateway connection string you had communicated to the SBManagementDB (using the New-SBFarm cmdlet) was pointing to the Old Server.
- When you removed the Node from the Farm and again Joined back - you removed one node from the configuration and re-added it - no affect :)
The ANSWER
Use Restore-SBFarm PS Cmdlet to communicate to the SBManagementDB that you changed the GW db
and then Use Restore-SBMessageContainer PS Cmdlet to communicate to Gateway DB that you changed the Container databases.
Now, add the Nodes back to this restored farm.
HTH!
Sree
In my current code base, when i create a VM, DNS name is being dynamically set as same as the instance name. For example, consider if my VM name is "anandInstance", DNS name of the name is being generated as "anandInstance.cloudapp.net". Is there a way to change the DNS name like "dns1.cloudapp.net" during the creation thru REST API??
"Connect to existing VM" , is it possible to achieve this option through REST call? In case "connect to existing.." option , we are getting a list of vms/services to choose and VM is getting created successfully. How to achieve the same using API.
Thanks
In my current code base, when i create a VM, DNS name is being
dynamically set as same as the instance name. For example, consider if
my VM name is "anandInstance", DNS name of the name is being generated
as "anandInstance.cloudapp.net". Is there a way to change the DNS name
like "dns1.cloudapp.net" during the creation thru REST API??
I don't think it is possible. Imagine what a nightmare in the portal would become if you were able to do so? How would you link a Cloud Service (whatever.cloudapp.net) to an actual deployment (MyDemoVm123). However you can use your own domain and have CNAME records pointing to your "want-to-change-for-some-reason.cloudapp.net" (frankly I surely think that soon we will use even longer names)
"Connect to existing VM" , is it possible to achieve this option
through REST call?
Connection to a VM is essentially opening a RDP session. If it a windows VM, you can try using the Download RDP file API call. Once you get the file, just start it with "process.start". If it is linux VM, just start SSH client on port 22 (or one you have defined) from the Cloud Service DNS name you have.
UPDATE
From the azure portal,for stand alone machineoption, we are able to give the dns name with deafult cloudoneapp.net. How to do the same
through the rest api call.any specfic paramter is there to specify the
same?
When you are using the REST API, you first create a Cloud Service (still named hosted service in the REST API) where your machine will be hosted. Here you give the name for that hosted service (the dns name with deafult cloudoneapp.net). Then you call the Create Virtual Machine Deployment API action.
In case "connect to existing.." option , we are getting a list of vms/services to choose and VM is getting created successfully. How to
achieve the same using API.
When you want to get list of all VMs, just get a list of all Hosted Services, then get properties of each and make a guess whether it is a VM or a Cloud Service (maybe by querying for Properties of each service). I don't see a direct access to the list of Virtual Machines. But as this feature being PREVIEW, things might change in the future.
Hope my answer is clear?