Passing service instance ID into guest executable at runtime - azure-service-fabric

Is it possible to pass service instance ID (int value) into guest executable at runtime? I have looked at <ExeHost><Arguments>, but it's only good for static data that has to be provided up front.

The environment variable provides the service package instance ID which is not the same as the instance/replica ID. Generally speaking, SF's environment variables can provide the same information available using FabricRuntime, i.e. the node context and the code package activation context. In native SF services, the instance ID is provided at run-time by the Fabric (in the ServiceContext class), as a single process can host multiple partitions and instances/replicas.
In a guest executable, which does not use SF APIs, the only option AFAIK is to query the Fabric for this information in a separate executable, run it as the SetupEntryPoint (which runs every time before the guest executable) and write the information to a file.
For example (compile the code into GetFabricData.exe and add it to the code package):
private static async Task MainAsync(string[] args)
{
var serviceTypeName = args.FirstOrDefault();
if (string.IsNullOrEmpty(serviceTypeName)) throw new ArgumentNullException(nameof(serviceTypeName));
using (var client = new FabricClient())
{
var activationContext = FabricRuntime.GetActivationContext();
var nodeContext = FabricRuntime.GetNodeContext();
var nodeName = nodeContext.NodeName;
var applicationName = new Uri(activationContext.ApplicationName);
var replicas = await client.QueryManager.GetDeployedReplicaListAsync(nodeName, applicationName);
// usually taking the first may not be correct
// but in a guest executable it's unlikely there would be multiple partitions/instances
var instance = replicas.OfType<DeployedStatelessServiceInstance>()
.FirstOrDefault(c => c.ServiceTypeName == serviceTypeName);
if (instance == null)
{
throw new InvalidOperationException($"Unable to find a service instance for {serviceTypeName}");
}
File.WriteAllText("FabricData", instance.InstanceId.ToString());
}
}
And in the service manifest:
<SetupEntryPoint>
<ExeHost>
<Program>GetFabricData.exe</Program>
<Arguments>Guest1Type</Arguments>
</ExeHost>
</SetupEntryPoint>
Then, the guest executable can simply read the FabricData file.

It's available in an environment variable. See here for a complete list of environment variables available to services: https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-manage-multiple-environment-app-configuration
Note that the one you're asking for, Fabric_ServicePackageInstanceId is generally only meant for internal consumption, and it identifies the entire service package. That means if you have multiple code packages (executables) in your service package, they will all get the same ID.

Related

How to get internal ip of postgreSQL DB in GCP created by Terraform

I am learning terraform deployments coupled with GCP to streamline deployments.
I have successfully deployed a postgreSQL db.
Now I am trying to utilize terraform outputs to write a the private ip generated by the postgreSQL DB server to the output directory where terraform is initiated from.
What is not clear to me is:
(1) The output is defined within the same main.tf file?
(2) Where is the output parameters referenced from? I cannot find the documentation to properly aline. Such I keep getting the error upon applying: Error: Reference to undeclared resource
My main.tf looks like this
resource "google_sql_database_instance" "main" {
name = "db"
database_version = "POSTGRES_12"
region = "us-west1"
settings {
availability_type = "REGIONAL"
tier = "db-custom-2-8192"
disk_size = "10"
disk_type = "PD_SSD"
disk_autoresize = "true"
}
}
output "instance_ip_addr" {
value = google_sql_database_instance.private_network.id
description = "The private IP address of the main server instance."
}
As for the code style, usually there would be a separate file called outputs.tf where you would add all the values you want to have outputted after a successful apply. The second part of the question is two-fold:
You have to understand how references to resource attributes/arguments work [1][2]
You have to reference the correct logical ID of the resource, i.e., the name you assigned to it, followed by the argument/attribute [3]
So, in your case that would be:
output "instance_ip_addr" {
value = google_sql_database_instance.main.private_ip_address # <RESOURCE TYPE>.<NAME>.<ATTRIBUTE>
description = "The private IP address of the main server instance."
}
[1] https://www.terraform.io/language/expressions/references#references-to-resource-attributes
[2] https://www.terraform.io/language/resources/behavior#accessing-resource-attributes
[3] https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/sql_database_instance#attributes-reference
To reference an attribute of a resource, you should put something like:
[resource type].[resource name].[attribute]
In this case, the output should be:
output "instance_ip_addr" {
value = google_sql_database_instance.main.private_ip_address
description = "The private IP address of the main server instance."
}
The output attributes are listed in the documentation. It's fine to put that in main.tf.

Does build pipeline requires internate connection to run

I am using Onprimise AzureDevOps. I am having intranet within the server but no access to internet.
Does the Use Nugget and Nugget installer requires internet to get it success.
If you look here you will find this code:
async function getLatestMatchVersionInfo(versionSpec: string): Promise<INuGetVersionInfo> {
taskLib.debug('Querying versions list');
let versionsUrl = 'https://dist.nuget.org/tools.json';
let proxyRequestOptions = {
proxy: taskLib.getHttpProxyConfiguration(versionsUrl)
};
let rest: restm.RestClient = new restm.RestClient('vsts-tasks/NuGetToolInstaller', undefined, undefined, proxyRequestOptions);
let nugetVersions: INuGetVersionInfo[] = (await rest.get<INuGetVersionInfo[]>(versionsUrl, GetRestClientOptions())).result;
// x.stage is the string representation of the enum, NuGetReleaseStage.Value = number, NuGetReleaseStage[NuGetReleaseStage.Value] = string, NuGetReleaseStage[x.stage] = number
let releasedVersions: INuGetVersionInfo[] = nugetVersions.filter(x => x.stage.toString() !== NuGetReleaseStage[NuGetReleaseStage.EarlyAccessPreview]);
let versionStringsFromDist: string[] = releasedVersions.map(x => x.version);
let version: string = toolLib.evaluateVersions(versionStringsFromDist, versionSpec);
if (!version)
{
taskLib.error(taskLib.loc("Error_NoVersionWasFoundWhichMatches", versionSpec));
taskLib.error(taskLib.loc("Info_AvailableVersions", releasedVersions.map(x => x.version).join("; ")));
throw new Error(taskLib.loc("Error_NuGetToolInstallerFailer", NUGET_TOOL_NAME));
}
return releasedVersions.find(x => x.version === version);
}
And it basically looks for nuget version if it will not find it on local machine. I assume that all this REST calls will not work without internet. So if you do not want to provide access to internet you should use only those versions installed on agents.
Agree with Krzysztof Madej, I think it is feasible to run build pipeline without internet connection. From the description of NuGet tool installer task we can know :
Acquires a specific version of NuGet from the internet or the tools
cache and adds it to the PATH.
So you need to make sure that the nuget version you need is installed on the local machine.
If you don't use upstream packages from the feed in your DevOps server, then your DevOps server machine could just in intranet network.
However, your build agent machine requires internet access, if you to do something that requires to access internet, such as Use Nuget task.

GCP Pub/Sub throws "The Application Default Credentials are not available"

I am trying to publish to Google Pub/Sub topic using the following:
ProjectTopicName topicName = ProjectTopicName.of("my-project-id", "my-topic-id");
Publisher publisher = null;
try {
// Create a publisher instance with default settings bound to the topic
publisher = Publisher.newBuilder(topicName).build();
List<String> messages = Arrays.asList("first message", "second message");
for (final String message : messages) {
ByteString data = ByteString.copyFromUtf8(message);
PubsubMessage pubsubMessage = PubsubMessage.newBuilder().setData(data).build();
// Once published, returns a server-assigned message id (unique within the topic)
ApiFuture<String> future = publisher.publish(pubsubMessage);
// Add an asynchronous callback to handle success / failure
ApiFutures.addCallback(
future,
new ApiFutureCallback<String>() {
#Override
public void onFailure(Throwable throwable) {
if (throwable instanceof ApiException) {
ApiException apiException = ((ApiException) throwable);
// details on the API exception
System.out.println(apiException.getStatusCode().getCode());
System.out.println(apiException.isRetryable());
}
System.out.println("Error publishing message : " + message);
}
#Override
public void onSuccess(String messageId) {
// Once published, returns server-assigned message ids (unique within the topic)
System.out.println(messageId);
}
},
MoreExecutors.directExecutor());
}
} finally {
if (publisher != null) {
// When finished with the publisher, shutdown to free up resources.
publisher.shutdown();
publisher.awaitTermination(1, TimeUnit.MINUTES);
}
}
I have changed the default values you see here to the particulars of the account I am hitting.
The environment variable points to the JSON file containing the pub/sub authentication credentials:
GOOGLE_APPLICATION_CREDENTIALS
was set using:
export GOOGLE_APPLICATION_CREDENTIALS=path/to/file.json
and verified with echo $GOOGLE_APPLICATION_CREDENTIALS - after a reboot.
But I am still encountering:
The Application Default Credentials are not available. They are available
if running in Google Compute Engine. Otherwise, the environment variable
GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining
the credentials. See https://developers.google.com/accounts/docs/application-
default-credentials for more information.
I believe this is related to the default environment that the application is running in, or rather what GCP object thinks the context is -runningOnComputeEngine:
com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
also, a dialog displayed:
Unable to launch App Engine Server
Cannot determine server execution context
and there are no Google Cloud Platform settings in project (Eclipse 2019-3):
This is not an App Engine application.
How to set the environment that GCP objects point to -> Non App Engine.
For reference:
Server to Server (link in error message)
Publish
Google Cloud Tools for Eclipse
Java 7 application
Mac OS (Sierra)
The file permissions are set that app can read the file.
Google's documentation on this is terrible - it does not mention this anywhere.
The answer is to use:
// create a credentials provider
CredentialsProvider credentialsProvider = FixedCredentialsProvider.create(ServiceAccountCredentials.fromStream(new FileInputStream(Constants.PUB_SUB_KEY)));
// apply credentials provider when creating publisher
publisher = Publisher.newBuilder(topicName).setCredentialsProvider(credentialsProvider).build();
The Environment variable usage is either deprecated or the documentation is flat out wrong, or I'm missing something,... which is entirely possible given the poor documentation.

Service Fabric - Are Endpoint definitions required for service remoting?

I'm trying to understand in what scenarios endpoint definitions are required in the ServiceManifest. I have a stateful service with the multiple service remoting listeners defined. My implementation of CreateServiceReplicaListeners:
protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
{
return new[]
{
new ServiceReplicaListener(context => this.CreateServiceRemotingListener(context)),
new ServiceReplicaListener(context =>
{
return new FabricTransportServiceRemotingListener(context,
new CustomService<string>(),
new FabricTransportRemotingListenerSettings
{
EndpointResourceName = "ustdfdomgfasdf"
});
}, name: "CustomListener")
};
}
The endpoint resource name for the custom listener is garbage. I have not defined that endpoint in the service manifest's resources:
<Resources>
<Endpoints>
<Endpoint Name="ServiceEndpoint" />
<Endpoint Name="ReplicatorEndpoint" />
</Endpoints>
</Resources>
However, in testing I find I'm still able to get a proxy to CustomListener:
InventoryItem i = new InventoryItem(description, price, number, reorderThreshold, max);
var applicationInstance = FabricRuntime.GetActivationContext().ApplicationName.Replace("fabric:/", String.Empty);
var inventoryServiceUri = new Uri("fabric:/" + applicationInstance + "/" + InventoryServiceName);
//Doesn't fail
ICustomService customServiceClient = ServiceProxy.Create<ICustomService>(inventoryServiceUri,
i.Id.GetPartitionKey(),
listenerName: "CustomListener");
//Still doesn't fail
var added = await customServiceClient.Add(1, 2);
To me, this indicates endpoint definitions aren't required for service remoting as long as the endpoint and listener names are unique. Is that so? If not, why does my example work?
Endpoints are required to tell service fabric to allocate ports in the node for the services being started on that node, this will prevent port collision when many services are opening ports in the node.
Once allocated, these are create as Environment Variables in the service process, something like: Fabric_Endpoint_<EndpointName> : port
When you create the Listeners, they are responsible to open the ports, generally using the ports allocated via Endpoints, but not prevents you creating a custom listener to Open any port (If running with enough privilege to do so)
CreateServiceRemotingListener(context) creates the default listeners
EndpointResourceName setting tell which endpoint to be used by a listener, if not defined, DefaultEndpointResourceName setting is used as the default Endpoint, the default value is "ServiceEndpoint"
What I am not sure to answer right now is: if EndpointResourceName is not found, it uses DefaultEndpointResourceName, I assume so, need to check the code to confirm that.
When multiple listeners are using the same port, they generally have a path to identify each of them, something like: tcp://127.0.0.1:1234/endpointpath

How to get started with ESAPI out of a servlet container

Could anyone give some considerations to get started using the ESAPI on a no-web context?
I came with this little test that validates a string with DefaultValidator.isValidCreditCard, but I got some web-container dependency errors.
The following method is consumed from a Junit Test:
#Override
public ValidationErrorList creditCard(String value) {
this.value = value;
ValidationErrorList errorList = new ValidationErrorList();
try {
isValid = validator.isValidCreditCard(null, value, false, errorList);
}catch(Exception ie){
System.out.println(">>> CCValidator: [ " + value + "] " + ie.getMessage());
messages = (ArrayList) errorList.errors();
}
return messages;
}
This is the error that I get (relevant part) of course I'm not running in a container:
Attempting to load ESAPI.properties via file I/O.
Attempting to load ESAPI.properties as resource file via file I/O.
Found in 'org.owasp.esapi.resources' directory: C:\foundation\validation\providers\esapi\ESAPI.properties
Loaded 'ESAPI.properties' properties file
Attempting to load validation.properties via file I/O.
Attempting to load validation.properties as resource file via file I/O.
Found in 'org.owasp.esapi.resources' directory: C:\foundation\validation\providers\esapi\validation.properties
Loaded 'validation.properties' properties file
SecurityConfiguration for Encoder.AllowMixedEncoding not found in ESAPI.properties. Using default: false
SecurityConfiguration for Encoder.AllowMixedEncoding not found in ESAPI.properties. Using default: false
javax/servlet/ServletRequest
java.lang.NoClassDefFoundError: javax/servlet/ServletRequest
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.owasp.esapi.util.ObjFactory.make(ObjFactory.java:74)
at org.owasp.esapi.ESAPI.httpUtilities(ESAPI.java:121)
at org.owasp.esapi.ESAPI.currentRequest(ESAPI.java:70)
at org.owasp.esapi.reference.Log4JLogger.log(Log4JLogger.java:434)
...
Calls to ESAPI..xxxMethods() also raise dependency errors.
Any advice to get started will be appreciate.
Best,
jose
ESAPI has a servlet filter API that requires javax.servlet.ServletRequest to be on the classpath. ESAPI is owned by OWASP --> "Open Web Application Security Project." Therefore, ESAPI is designed with web applications in mind.
If you're not writing a web application, then its either a console application or a rich client application. If you don't expect to use it to connect to the outside world, then the main secure practices you really need to worry about are ensuring that you always use safely parameterized queries, and that any data passed into your application from a source that IS connected to the outside world is properly escaped. For that, the only thing you need is OWASP's encoder project.