RabbitMQ failed to start, TCP connection succeeded but Erlang distribution failed - server

I'm a new one just start to learn and install RabbitMQ on Windows System.
I install Erlang VM and RabbitMQ in custom folder, not default folder (Both of them).
Then I have restarted my computer.
By the way,My Computer name is "NULL"
I cd to the RabbitMQ/sbin folder and use command:
rabbitmqctl status
But the return message is:
Status of node rabbit#NULL ...
Error: unable to perform an operation on node 'rabbit#NULL'.
Please see diagnostics information and suggestions below.
Most common reasons for this are:
Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
Target node is not running
In addition to the diagnostics info below:
See the CLI, clustering and networking guides on http://rabbitmq.com/documentation.html to learn more
Consult server logs on node rabbit#NULL
DIAGNOSTICS
attempted to contact: [rabbit#NULL]
rabbit#NULL:
connected to epmd (port 4369) on NULL
epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic
TCP connection succeeded but Erlang distribution failed
Authentication failed (rejected by the remote node), please check the Erlang cookie
Current node details:
node name: rabbitmqcli70#NULL
effective user's home directory: C:\Users\Jerry Song
Erlang cookie hash: 51gvGHZpn0gIK86cfiS7vp==
I have try to RESTART RabbitMQ, What I get is:
ERROR: node with name "rabbit" already running on "NULL"
By the way,My Computer name is "NULL"
And I have enable all ports in firewall.

https://groups.google.com/forum/#!topic/rabbitmq-users/a6sqrAUX_Fg
describes the problem where there is a cookie mismatch on a fresh installation of Rabbit MQ. The easy solution on windows is to synchronize the cookies
Also described here: http://www.rabbitmq.com/clustering.html#erlang-cookie
Ensure cookies are synchronized across 1, 2 and Optionally 3 below
%HOMEDRIVE%%HOMEPATH%\.erlang.cookie (usually C:\Users\%USERNAME%\.erlang.cookie for user %USERNAME%) if both the HOMEDRIVE and HOMEPATH environment variables are set
%USERPROFILE%\.erlang.cookie (usually C:\Users\%USERNAME%\.erlang.cookie) if HOMEDRIVE and HOMEPATH are not both set
For the RabbitMQ Windows service - %USERPROFILE%\.erlang.cookie (usually C:\WINDOWS\system32\config\systemprofile)
The cookie file used by the Windows service account and the user running CLI tools must be synchronized by copying the one from C:\WINDOWS\system32\config\systemprofile folder.

If you are using dedicated drive folder locations for your development tools/software in Windows10(Not the windows default location), one way you can synchronize the erlang cookie as described by https://www.rabbitmq.com/cli.html is by copying the cookie as explained below.
Please note in my case HOMEDRIVE and HOMEPATH environment variables both are not set.
After copying the "C:\Windows\system32\config\systemprofile\.erlang.cookie" to "C:\Users\%USERNAME%\.erlang.cookie" ,
the error "tcp connection succeeded but Erlang distribution failed" is resolved.
Now I am able to use "rabbitmqctl.bat status" command successfully. Hence there is no mandatory need to install in default location to resolve this error as synchronizing cookie will resolve that error.

In my case similar issue (Authentication failed because of Erlang cookies mismatch) solved by copying .erlang.cookie file from Windows system dir - C:\Windows\system32\config\systemprofile\.erlang.cookie to %HOMEDRIVE%%HOMEPATH%\.erlang.cookie (where %HOMEDRIVE% was set to H: and %HOMEPATH% to \ respectively)
Quick setup TODO for Windows, Erlang OTP 24 and RabbitMQ 3.8.19:
Download & Install Erlang [OTP 24] (needs Admin rights) from:
https://www.erlang.org/downloads
set ERLANG_HOME (should point to install dir)
Download & Install recent [3.8.19] RabbitMQ (needs Admin rights) from:
https://github.com/rabbitmq/rabbitmq-server/releases/
Follow: https://www.rabbitmq.com/install-windows.html and/or
https://www.rabbitmq.com/install-windows-manual.html
set RABBITMQ_SERVER (should point to install dir)
update %PATH% by adding: ;%RABBITMQ_SERVER%\sbin
Fix Erlang-cookie issue from above, follow: https://www.rabbitmq.com/cli.html#erlang-cookie
Enable Web UI by running command: %RABBITMQ_SERVER%/sbin/rabbitmq-plugins.bat enable rabbitmq_management
From item #8 (above) got error because of missing file: %USERPROFILEDIR%/AppData/Roaming/RabbitMQ/enabled_plugins -> had to create it and run %RABBITMQ_SERVER%/sbin/rabbitmq-plugins.bat enable rabbitmq_management again!
Run/restart on the way might be required
Finally, login to: http://localhost:15672/ (guest:guest)
, or check by cURL:
curl -i -u guest:guest http://localhost:15672/api/vhosts
should receive response like:
HTTP/1.1 200 OK
cache-control: no-cache
content-length: 186
content-security-policy: script-src 'self' 'unsafe-eval' 'unsafe-inline';
object-src 'self'
content-type: application/json
date: Tue, 13 Jul 2021 11:21:12 GMT
server: Cowboy
vary: accept, accept-encoding, origin
[{"cluster_state":{"rabbit#hostname":"running"},"description":"Default virtual host","metadata":{"description":"Default virtual host","tags":[]},"name":"/","tags":[],"tracing":false}]
P.S. Some useful RabbitMQ CLI commands (copy-paste):
%RABBITMQ_SERVER%/sbin/rabbitmqctl start_app
%RABBITMQ_SERVER%/sbin/rabbitmqctl stop_app
%RABBITMQ_SERVER%/sbin/rabbitmqctl status
P.P.S. UPDATE: great article for this subject: https://www.journaldev.com/11655/spring-rabbitmq

I have reinstalled the RabbitMQ in my computer by using default setup folder
Then checked with the command :
rabbitmqctl status
It works now, not the problem of Erlang VM .(Means Er can install at another folder)
It will cause some problem (like this one) that I couldn't find out now if we don't use the RabbitMQ default setup require folder (C:\Program Files\RabbitMQ Server)
If anyone finds it out, I hope you can tell me why and how to fix.

How I resolved mine
It's mostly caused by cookie mismatch on a fresh installation of Rabbit MQ
follow this 2 steps
1. copy the .erlang.cookie file from C:\Windows\System32\config\systemprofile paste it into
C:\Users\["your user nameusername"] folder
2. run rabbitmq-service.bat stop and rabbitmq-service.bat start
Done it should work now when you run 'rabbitmqctl start_app' good luck.
note if you have more than one user put it in the correct user folder

In Centos.
add ip nodename pair to /etc/hosts on each node.
restart rabbitmq-server service on each slave node.
works for me.

i got error like this, i just stop my rabbitMQ with close port 25672
here syntax for linux:
kill -9 $(lsof -t -i:25672)
Error Images:

Just adding my experience if it helps others down the line.
I wrote a Powershell .ps1 script to install and configure RabbitMQ which would be used as one of the stept to provision a server with Packer.
I wrote the code on a fresh AWS W2016 Server build. It worked fine when run on the box (as administrator, from an admin PS console) but when the same code was moved over to the Packer build server, it would fall over when doing the rabbitmqctl.bat configuration steps via packer, despite both using (as far as I can tell) Administrator to run the scripts.
So this worked on the coding box:
$pathvargs = {cmd.exe /c "rabbitmqctl.bat" add_user Username Password}
Invoke-Command -ScriptBlock $pathvargs
$pathvargs = {cmd.exe /c "rabbitmqctl.bat" set_user_tags User administrator}
Invoke-Command -ScriptBlock $pathvargs
$pathvargs = {cmd.exe /c "rabbitmqctl.bat" set_permissions -p "/" User "^User-.*" ".*" ".*"}
Invoke-Command -ScriptBlock $pathvargs
Write-Host "Did RabbitMQ"
But I had to prelude this with...
copy "C:\Windows\system32\config\systemprofile\.erlang.cookie" "C:\Program Files\RabbitMQ Server\rabbitmq_server-3.7.17\sbin\.erlang.cookie"
copy "C:\Windows\system32\config\systemprofile\.erlang.cookie" $env:userprofile\.erlang.cookie -force
... On the Packer box.
I am guessing there is some context issue going on but I'm using
"winrm_username": "Administrator",
in the Packer builders block, so I thought this would suffice.
TL;DR - Use the Cookie even though it works without it in some instances.

I have encountered the same error after installing Erlang VM and RabbitMQ using the default installation folders in Windows 10. Managed to start the management and access it via HTTP, but status failed with this error.
The cookie was fine in all folders (%HOMEDRIVE%%HOMEPATH%, %USERPROFILE%, C:\WINDOWS\system32\config\systemprofile).
I had to perform a restart the Windows to make it work. After restart it set up something to run at startup + asked permission to make an exception in the firewall.

In my case, the file was at c:\\Windows\.erlang.cookie, just copied it to C:\Users{USERNAME} and all works, thanks to everyone for the hits

Another thing to check after making sure the cookie file is in all the locations.. is to realize that you installed 32 bit Erlang.. not 64..
Happened to me. Removed 32 bit Erlang and Installed 64 and rabbitmqctl status returns expected results.

Related

Failure/timeout invoking Lambda locally with SAM

I'm trying to get a local env to run/debug Python Lambdas with VSCode (windows). I'm using a provided HelloWorld example to get the hang of this but I'm not being able to invoke.
Steps used to setup SAM and invoke the Lambda:
I have Docker installed and running
I have installed the SAM CLI
My AWS credentials are in place and working
I have no connectivity issues and I'm able to connect to AWS normally
I create the SAM application (HelloWorld) with all the files and resources, I didn't change anything.
I run "sam build" and it finishes sucessfully
I run "sam local invoke" and it fails with timeout. I increased the timeout to 10s, still times out. The HelloWorld Lambda code only prints and does nothing else, so I'm guessing the code isn't the problem, but something else relating to the container or the SAM env itself.
C:\xxxxxxx\lambda-python3.8>sam build Your template contains a
resource with logical ID "ServerlessRestApi", which is a reserved
logical ID in AWS SAM. It could result in unexpected behaviors and is not recommended.
Building codeuri:
C:\xxxxxxx\lambda-python3.8\hello_world runtime: python3.8 metadata:
{} architecture: x86_64 functions: ['HelloWorldFunction'] Running
PythonPipBuilder:ResolveDependencies Running
PythonPipBuilder:CopySource
Build Succeeded
Built Artifacts : .aws-sam\build Built Template :
.aws-sam\build\template.yaml
C:\xxxxxxx\lambda-python3.8>sam local invoke Invoking
app.lambda_handler (python3.8) Skip pulling image and use local one:
public.ecr.aws/sam/emulation-python3.8:rapid-1.51.0-x86_64.
Mounting C:\xxxxxxx\lambda-python3.8.aws-sam\build\HelloWorldFunction
as /var/task:ro,delegated inside runtime container Function
'HelloWorldFunction' timed out after 10 seconds
No response from invoke container for HelloWorldFunction
Any hints on what's missing here?
Thanks.
Mostly, a lambda function gets timed out because of some resource dependency. Are you using any external resource, maybe db connection or some REST API call ?
Please put more prints in lambda_handler(your function handler), before calling any resource, then you might know where exactly it is waiting. Also increase the timeout to 1 minute or more because most of the external resource call over HTTPS will have 30 secs timeouts.
The log suggests that either the container wasn't started, or SAM couldn't connect to it.
Sometimes the hostname resolution on Windows can be affected by hosts file or system settings.
Try running the invoke command as follows (this will make the container ports bind to all interfaces):
sam local invoke --container-host-interface 0.0.0.0
...additionally try setting the container-host parameter (set to localhost by default):
sam local invoke --container-host-interface 0.0.0.0 --container-host host.docker.internal
The next piece of puzzle is incorporating these settings into VSCODE. This can to be done in two places:
create samconfig.toml in the root dir of the project with the following contents. This will allow running sam local invoke from the terminal without having to add the command line argument:
version=0.1
[default.local_invoke.parameters]
container_host_interface = "0.0.0.0"
update launch configuration as follows to enable VSCode debugging:
...
"sam": {
"localArguments": ["--container-host-interface","0.0.0.0"]
}
...

Does Server -dev mode store the data in windows

I tried running the vault server in local with dev mode option. I got a root token which i exported to the environment variables. But once I stopped the server and started it it said *Invalid Request, Unable to start the server with the token my token.
Also does the in-memory vault server store its secrets?
If so where does it store secrets in my windows machine? I have exported VAULT_DEV_ROOT_TOKEN_ID to my environment variables with value s.WC4LYVf6oOyllP6HjR0A3nvo
I tried restarting the server several times
C:\Users\user>vault server -dev
==> Vault server configuration:
Api Address: http://127.0.0.1:8200
Cgo: disabled
Cluster Address: https://127.0.0.1:8201
Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Log Level: info
Mlock: supported: false, enabled: false
Storage: inmem
Version: Vault v1.2.2
Error initializing Dev mode: failed to create root token with ID "s.WC4LYVf6oOyllP6HjR0A3nvo": 1 error occurred:
* invalid request
The problem here was when ever we start a the vault server in dev mode in windows it generates a root access token. If we export the VAULT_DEV_ROOT_TOKEN_ID to the environment variables, it tries to start the server with that token. But since the token was previously used by vault it the server is not allowed to start.
Because you are running on the dev mode you'll need to unset the token, because whenever you restart the server it generates a new token, and invalidates the previous one. This is how to get it done on MacOS, specifically Big Sur.
Do these:
Open the .zshrc file. Issue this command in the terminal open ~/.zshrc to open it.
In the previous setup, the vault url and token has been added to .zshrc, comment these lines out. These are the lines:
export VAULT_ADDR='http://127.0.0.1:8200
export VAULT_DEV_ROOT_TOKEN_ID=s.a009oZnrl78a9h1vlRw1kGqL
Issue the startup command on the terminal again - vault server -dev
This time around it'll generate a new token, copy it, paste, and replace the previous token. This is the one to edit and replace -
export VAULT_DEV_ROOT_TOKEN_ID=s.a009oZnrl78a9h1vlRw1kGqL.
Once you've added it, uncomment these vault values in the .zshrc file.
Save the file and close, then on your terminal issue this command - source ~/.zshrc to refresh/update the .zshrc file.

installing kubernetes on coreos with rkt and automated script

I'm trying to install kuberentes with rkt on my real (not virtual) coreos servers at home using the scripts at https://github.com/coreos/coreos-kubernetes/tree/master/multi-node/generic and I have some questions.
my etcd2 is using tls keys, I can't see anywhere in the script where I can define where the certificates are located.
can I supply a domain instead of IP for ADVERTISE_IP and CONTROLLER_ENDPOINT ?
when I tried to install kubernetes manually I needed start the rkt service api. it doesn't state in the documents that it needed here, does it mean that I don't need it if I use these scripts? or is it just something that's missing in the documents?
thanks!
update
Rob thank you so much for your response. I wasn't clear enough regarding etcd2. I already have etcd2 tls installed and properly configured on my coreos servers. so I configured my etcd servers in the controller-install.sh file:
export ETCD_ENDPOINTS="https://coreos-2.tux-in.com:2379,https://coreos-3.tux-in.com:2379"
but when I run the controller-install.sh script, it returns and repeat the following output:
Waiting for etcd...
Trying: https://coreos-2.tux-in.com:2379
Trying: https://coreos-3.tux-in.com:2379
Trying: https://coreos-2.tux-in.com:2379
Trying: https://coreos-3.tux-in.com:2379
...
so I was guessing it's because i didn't define etcd related tls certificates in the controller script and that is why it stuck in that faze.
on my macbook pro laptop I have the following alias configured:
alias myetcdctl="~/apps/etcd-v3.0.8-darwin-amd64/etcdctl --endpoint=https://coreos-2.tux-in.com:2379 --ca-file=/Users/ufk/Projects/coreos/tux-in/etcd/certs/certs-names/ca.pem --cert-file=/Users/ufk/Projects/coreos/tux-in/etcd/certs/certs-names/etcd1.pem --key-file=/Users/ufk/Projects/coreos/tux-in/etcd/certs/certs-names/etcd1-key.pem --timeout=10s"
so when I run myetcdctl member list I get:
8832ce6a269a7dac: name=ccff826d5f564c67abf35467306f80a0 peerURLs=https://coreos-3.tux-in.com:2380 clientURLs=https://coreos-3.tux-in.com:2379 isLeader=true
a2c0ac9708ef90fc: name=dc38bc8f20e64940b260d3f7b260430d peerURLs=https://coreos-2.tux-in.com:2380 clientURLs=https://coreos-2.tux-in.com:2379 isLeader=false
so I'm guessing that I don't really have a problem there.
any ideas?
thanks!
my etcd2 is using tls keys, I can't see anywhere in the script where I can define where the certificates are located.
These scripts don't start an etcd server. You will need to set one up manually and will be able to use TLS and as many nodes as you would like. This isn't clear in the current form of the document, I will attempt a PR to fix.
can I supply a domain instead of IP for ADVERTISE_IP and CONTROLLER_ENDPOINT ?
Only CONTROLLER_ENDPOINT be a domain name.
when I tried to install kubernetes manually I needed start the rkt service api. it doesn't state in the documents that it needed here, does it mean that I don't need it if I use these scripts? or is it just something that's missing in the documents?
These scripts include/start the rkt API service. As you can see below, it also has a Restart parameter set (source):
[Unit]
Before=kubelet.service
[Service]
ExecStart=/usr/bin/rkt api-service
Restart=always
RestartSec=10
[Install]
RequiredBy=kubelet.service

ejabberd on openshift - Failed RPC connection to the node

I am trying to configure ejabberd on DIY cartridge on openshift, following the guide here:
Erlang and Ejabberd on OpenShift
I followed everything successsfully up to here:
Next you can start ejabberd running the following 2 commands, which you’ll want to put in your .openshift/action_hooks/start script
there is no error and ejabberd seems to be started, but the next command:
$OPENSHIFT_DATA_DIR/erl_home/sbin/ejabberdctl register admin localhost password1234
failed with this error:
Failed RPC connection to the node ‘ejabberd#127.7.131.1′: {‘EXIT’, {badarg, [{ets,lookup, [local_config, ejabberdctl_access_commands], []}, {ejabberd_config, get_local_option, 1, [{file, "ejabberd_config.erl"}, {line,590}]}, {ejabberd_ctl, get_accesscommands, 0, [{file, "ejabberd_ctl.erl"}, {line,236}]}, {ejabberd_ctl, process,1, [{file, "ejabberd_ctl.erl"}, {line,199}]}, {rpc, ‘-handle_call_call/6-fun-0-’, 5, [{file, "rpc.erl"}, {line,205}]}]}}
Commands to start an ejabberd node: start Start an ejabberd node in server mode debug Attach an interactive Erlang shell to a running ejabberd node live Start an ejabberd node in live (interactive) mode
Optional parameters when starting an ejabberd node: –config-dir dir Config ejabberd: /var/lib/openshift/52c9674d5973ca7734000180/app-root/data//erl_home/etc/ejabberd –config file Config ejabberd: /var/lib/openshift/52c9674d5973ca7734000180/app-root/data//erl_home/etc/ejabberd/ejabberd.cfg –ctl-config file Config ejabberdctl: /var/lib/openshift/52c9674d5973ca7734000180/app-root/data//erl_home/etc/ejabberd/ejabberdctl.cfg –logs dir Directory for logs: /var/lib/openshift/52c9674d5973ca7734000180/app-root/data//erl_home/var/log/ejabberd –spool dir Database spool dir: /var/lib/openshift/52c9674d5973ca7734000180/app-root/data//erl_home/var/lib/ejabberd –node nodename ejabberd node name: ejabberd#127.7.131.1
I am not sure what causes the error... it seems like it is trying to connect to localhost (due to the nodename: ejabberd#127.7.131.1). However, I have sed every localhost smell from the previous commands on the blog.
Anybody ever encounter this before? Any clue on how to debug is also highly appreciated, as I am not very familiar with openshift or ejabberd as well as linux... Thank you very much in advance!
I wrote the blog post, and saw this error on a new DIY Application I created. It was due to a bug with erlang and the openssl package in the openshift box. I put in a patch in the erlang source so that new compilations will work.
The bug that was causing the issue was here: https://bugzilla.redhat.com/show_bug.cgi?id=1023017
You'll need to run using your host instead of localhost.
$OPENSHIFT_DATA_DIR/erl_home/sbin/ejabberdctl register admin <replacewithyourhost> password1234
For that
OPENSHIFT_DATA_DIR/erl_home/sbin/ejabberdctl register admin $OPENSHIFT_DIY_IP password1234
However it returns error as well:
{error_logger,{{2014,1,13},{21,11,24}},"Protocol: ~tp: register/listen error: ~tp~n",["inet_tcp",econnrefused]} {error
above
You need to kill all the ejabberd process including epmd.
I had this problem one time, My solution was to edit /etc/hosts(for windows- C:\Windows\System32\drivers\etc\hosts) file and make sure that there was a hostname entry for my public ip address and the domain I wanted ejabber to respond to.
0.0.0.0 hostname.domain.com hostname
1.1.1.1(your ip) your-hostname.your-domain your-hostname

Nuget + NuGetPowerTools + Enable-PackageRestore = authentication failure behind Proxy

Last night, at home, where I do not work with a proxy, updated a NuGet repository using NuGetPowerTools / Enable-PackageRestore.
Worked great.
Came into the office, where i am working with the same Repository under the following conditions:
in a VMWare based VM,
using Bridged Networking,
Reinstalled NuGet today, restarted (to be sure I have latest version)
runnning under an account in a DEV Domain (ie, DEV\Me)
Accessing the outer world via a Proxy that requires a corp Domain Account (ie, CORP\Me)
Ie 9's Configuration/Internet Options/Connection is setup as:
Use Proxy:
Address: yadayada1
Port: 80
Bypass proxy for local addresses
Automatically detect Settings
Above settings are correct in as much that I can access the web via IE9, Chrome, etc.
NOTE: Can list and download Nuget packages...it's just the new Build process that can't.
As per suggestion on the web I have looked at DevEnv.exe.config and have the following settings:
-<system.net><settings><ipv6 enabled="true" /></settings></system.net>
I also tried with these settings as
<system.net><defaultProxy useDefaultCredentials="true"><proxy autoDetect="True" usesystemdefault="True" /></defaultProxy></system.net>
so that DevEnv.exe falls back for to same connection settings that IE 9 is using. Didn't do it.
What I get for all this is:
------ Build started: Project: XAct.Core, Configuration: Debug Any CPU ------
Please provide proxy credentials:
UserName: Password:
C:_Workspaces\XAct\CS.FF.XAct.Lib2.nuget\NuGet.targets(43,9): error : Cannot read keys > when either application does not have a console or when console input has been redirected > from a file. Try Console.Read.
C:_Workspaces\XAct\CS.FF.XAct.Lib2.nuget\NuGet.targets(43,9): error MSB3073: The
command ""C:_Workspaces\XAct\CS.FF.XAct.Lib2.nuget\nuget.exe" install
"C:_Workspaces\XAct\CS.FF.XAct.Lib2\XAct.Core\XAct.Core\packages.config" -source "" -o
"C:_Workspaces\XAct\CS.FF.XAct.Lib2\packages"" exited with code 1.
Found (but unfortunately lost again) a thread somewhere on the net where I saw NuGet developers thrashing it out -- referring to downloading the latest build from their TeamCity (which I could not access/join) -- but the thread did not clearly say Eureka! at the bottom.
Thanks.
Reiterating what Sky and Pranav mentioned - this issue seems to be fixed in NuGet 1.7 - see http://nuget.codeplex.com/workitem/1655:
pranavkm wrote Jan 31 at 8:17 PM
Fixed in changeset 6c156e0dc214