Does build pipeline requires internate connection to run - azure-devops

I am using Onprimise AzureDevOps. I am having intranet within the server but no access to internet.
Does the Use Nugget and Nugget installer requires internet to get it success.

If you look here you will find this code:
async function getLatestMatchVersionInfo(versionSpec: string): Promise<INuGetVersionInfo> {
taskLib.debug('Querying versions list');
let versionsUrl = 'https://dist.nuget.org/tools.json';
let proxyRequestOptions = {
proxy: taskLib.getHttpProxyConfiguration(versionsUrl)
};
let rest: restm.RestClient = new restm.RestClient('vsts-tasks/NuGetToolInstaller', undefined, undefined, proxyRequestOptions);
let nugetVersions: INuGetVersionInfo[] = (await rest.get<INuGetVersionInfo[]>(versionsUrl, GetRestClientOptions())).result;
// x.stage is the string representation of the enum, NuGetReleaseStage.Value = number, NuGetReleaseStage[NuGetReleaseStage.Value] = string, NuGetReleaseStage[x.stage] = number
let releasedVersions: INuGetVersionInfo[] = nugetVersions.filter(x => x.stage.toString() !== NuGetReleaseStage[NuGetReleaseStage.EarlyAccessPreview]);
let versionStringsFromDist: string[] = releasedVersions.map(x => x.version);
let version: string = toolLib.evaluateVersions(versionStringsFromDist, versionSpec);
if (!version)
{
taskLib.error(taskLib.loc("Error_NoVersionWasFoundWhichMatches", versionSpec));
taskLib.error(taskLib.loc("Info_AvailableVersions", releasedVersions.map(x => x.version).join("; ")));
throw new Error(taskLib.loc("Error_NuGetToolInstallerFailer", NUGET_TOOL_NAME));
}
return releasedVersions.find(x => x.version === version);
}
And it basically looks for nuget version if it will not find it on local machine. I assume that all this REST calls will not work without internet. So if you do not want to provide access to internet you should use only those versions installed on agents.

Agree with Krzysztof Madej, I think it is feasible to run build pipeline without internet connection. From the description of NuGet tool installer task we can know :
Acquires a specific version of NuGet from the internet or the tools
cache and adds it to the PATH.
So you need to make sure that the nuget version you need is installed on the local machine.

If you don't use upstream packages from the feed in your DevOps server, then your DevOps server machine could just in intranet network.
However, your build agent machine requires internet access, if you to do something that requires to access internet, such as Use Nuget task.

Related

Hosting a web server in a local enviroment using only Flutter

Is it possible to host a Flutter web app on a local environment using a Flutter desktop-based app?
The google-search for a solution like this can be difficult, since it involves many keywords that lead to similar situations (online hosting when you need a local solution, command-line only solution, and so on).
After some digging, I ended up using the shelf package to deploy my own Flutter web app on a local network. I developed this for Windows only, so I can't guarantee it will work on other platforms.
First thing to do is obviously adding the shelf package in your pubspec.yaml: after that, this is how my main method looks like
import 'package:shelf/shelf_io.dart' as shelf_io;
import 'package:shelf/shelf.dart' as shelf;
import 'package:shelf_router/shelf_router.dart' as shelf_router;
[...]
void main() async{
[...]
var secureContext = SecurityContext();
try {
//privKey and cert are the String names of the two files for the SSL connection,
//placed in the root directory of the flutter project or along with the .exe file (when released)
secureContext.usePrivateKey(privKey);
secureContext.useCertificateChain(cert);
} catch (error) {
logger.e("Error on init SecurityContext");
}
try {
//this is the handler that deploys the files contained in 'webAppFolder': I just simply pasted the result of
//the flutter webapp building inside (the index.html file is the default one for flutter web)
//and put the folder in the root of the flutter project (or, again, in the same folder with the .exe file when released)
final _staticHandler = createStaticHandler("webAppFolder", defaultDocument: 'index.html');
//this I kept just for a reminder on how to deploy a static page, if needed
final _router = shelf_router.Router()
..get(
'/time',
(request) => shelf.Response.ok(DateTime.now().toUtc().toIso8601String()),
);
final cascade = shelf.Cascade()
.add(_staticHandler)
.add(_router);
try {
var server = await shelf_io.serve(
cascade.handler,
InternetAddress.anyIPv4,
mainPort, //this is the number of the port on which the webapp is deployed (I load this from a .ini file beforehand
securityContext: secureContext,
);
// Enable content compression
server.autoCompress = true;
logger.i("Serving at https://${server.address.host}:${server.port}");
} catch (err) {
logger.e("Error while serving");
logger.e(err.toString());
}
} catch (err) {
logger.e("Error while creating handler");
logger.e(err.toString());
}
runApp(MaterialApp(
[...]
This is the part related to the deploy of a web app: since the flutter desktop app already provides a GUI, I used that to add some maintenance and testing utilities to check if everything is working fine.
For more details regarding shelf, refer to their API on their pub.dev page.

Does SBT to support preemptive auth for downloading packages?

I am running SBT 1.2.8 and my project needs to download packages from a repo on a privately hosted Artifactory instance. My repo is protected by basic auth. After reading a multitude of examples and instructions, I created a credentials.properties file in my repo.
realm=Artifactory Realm
host=artifactory.mycompany.com
username=my_username
password=my_password
I then added the following to my build.sbt file
credentials += Credentials(new File("credentials.properties"))
Then I added the repository to my list of resolvers in resolvers.sbt
"My Company Artifactory" at "https://artifactory.mycompany.com/artifactory/my_private_repo/",
I built my application and was able to download the protected packages just fine.
However, a system administrator at my company requested I turn on the “Hide Existence of Unauthorized Resources” setting in Artifactory. This setting forces Artifactory to return 404 errors when an unauthenticated user tries to access protected resources. Usually in this case, Artifactory returns 401s with a WWW-Authenticate header.
Suddenly, my application was unable to resolve its dependencies. I turned the Artifactory setting off and then back on again and verified this setting was, in fact, the cause of my problems.
It appears as though SBT will not send credentials unless it is challenged with a 401 and a WWW-Authenticate header (with the proper realm). Looking at the docs and GitHub issues for SBT, Ivy, and Coursier, it seems this “preemptive authentication” is not a supported feature.
I spend many hours trying to resolve this in various ways, but I cannot find a solution. Here is what I have tried:
Adding my Artifactory username and password to the repository url, so it looks like https://my_username:my_password#artifactory.mycompany.com/artifactory/my_private_repo/. This worked in my browser and a REST client, but not with SBT.
Omitting the “realm” from my credentials file
Switching to SBT 1.3.9 and trying everything above with the new version.
Does anyone know how I can get SBT to use preemptive HTTP basic auth? It appears both Maven and Gradle support this (see links below), but I cannot find anything in the SBT docs.
Maven support for preemptive auth: https://jfrog.com/knowledge-base/why-does-my-maven-builds-are-failing-with-a-404-error-when-hide-existence-of-unauthorized-resources-is-enabled/
Gradle support for preemptive auth:
https://github.com/gradle/gradle/pull/386/files
I'm almost thinking of setting up a local proxy to send the proper headers Artifactory, and point SBT to use the local proxy as a resolver. However, that seems needlessly cumbersome for developers to use.
you are correct.
You can setup an AbstractRepository. See https://github.com/SupraFii/sbt-google-artifact-registry/blob/master/src/main/scala/ch/firsts/sbt/gar/ArtifactRegistryRepository.scala#L21 for example:
package ch.firsts.sbt.gar
import java.io.File
import java.util
import com.google.cloud.artifactregistry.wagon.ArtifactRegistryWagon
import org.apache.ivy.core.module.descriptor.Artifact
import org.apache.ivy.plugins.repository.AbstractRepository
import org.apache.maven.wagon.repository.Repository
class ArtifactRegistryRepository(repositoryUrl: String) extends AbstractRepository {
val repo = new Repository("google-artifact-registry", repositoryUrl)
val wagon = new ArtifactRegistryWagon()
override def getResource(source: String): ArtifactRegistryResource = {
val plainSource = stripRepository(source)
wagon.connect(repo)
ArtifactRegistryResource(repositoryUrl, plainSource, wagon.resourceExists(plainSource))
}
override def get(source: String, destination: File): Unit = {
val adjustedSource = if (destination.toString.endsWith("sha1"))
source + ".sha1"
else if (destination.toString.endsWith("md5"))
source + ".md5"
else
source
wagon.connect(repo)
wagon.get(adjustedSource, destination)
}
override def list(parent: String): util.List[String] = sys.error("Listing repository contents is not supported")
override def put(artifact: Artifact, source: File, destination: String, overwrite: Boolean): Unit = {
val plainDestination = stripRepository(destination)
wagon.connect(repo)
wagon.put(source, plainDestination)
}
private def stripRepository(fullName: String): String = fullName.substring(repositoryUrl.length + 1)
}

Cannot access https://dev.azure.com/<myOrg> Using TFS extended client version 15

We are migrating some code that used to run against an on premise TFS server but now needs to run against Azure DevOps (previously Team Services). The credentials I'm using have been validated to successfully authenticate to our DevOps organization instance, but running the following code after referencing the
Microsoft.TeamFoundationServer.ExtendedClient
NuGet package always results in TF30063: You are not authorized to access https://dev.azure.com/<myOrg> The code is posted below for authenticating via non-interactive authentication. Do I need to use a different authentication mechanism or different credentials type to get this working?
System.Net.NetworkCredential networkCredential = new System.Net.NetworkCredential(_userName, DecryptedPassword, _domain);
try
{
// Create TeamFoundationServer object
_teamFoundationCollection = new TfsTeamProjectCollection(_serverUrl, networkCredential);
_teamFoundationCollection.Authenticate();
}
catch (Exception ex)
{
// Not authorized
throw new TeamFoundationServerException(ex.Message, ex.InnerException)
}
Since you want to use .Net Client Libraries, you could refer to the following link:
https://learn.microsoft.com/en-us/azure/devops/integrate/concepts/dotnet-client-libraries?view=azure-devops
Patterns for use:
using Microsoft.VisualStudio.Services.Common;
using Microsoft.VisualStudio.Services.Client;
using Microsoft.TeamFoundation.SourceControl.WebApi;
using Microsoft.VisualStudio.Services.WebApi;
const String c_collectionUri = "https://dev.azure.com/fabrikam";
const String c_projectName = "MyGreatProject";
const String c_repoName = "MyRepo";
// Interactively ask the user for credentials, caching them so the user isn't constantly prompted
VssCredentials creds = new VssClientCredentials();
creds.Storage = new VssClientCredentialStorage();
// Connect to Azure DevOps Services
VssConnection connection = new VssConnection(new Uri(c_collectionUri), creds);
// Get a GitHttpClient to talk to the Git endpoints
GitHttpClient gitClient = connection.GetClient<GitHttpClient>();
// Get data about a specific repository
var repo = gitClient.GetRepositoryAsync(c_projectName, c_repoName).Result;

Unable to download embedded MongoDB, behind proxy, using automatic configuration script

I have a Spring Boot project, built using Maven, where I intend to use embedded mongo db. I am using Eclipse on Windows 7.
I am behind a proxy that uses automatic configuration script, as I have observed in the Connection tab of Internet Options.
I am getting the following exception when I try to run the application.
java.io.IOException: Could not open inputStream for https://downloads.mongodb.org/win32/mongodb-win32-i386-3.2.2.zip
at de.flapdoodle.embed.process.store.Downloader.downloadInputStream(Downloader.java:131) ~[de.flapdoodle.embed.process-2.0.1.jar:na]
at de.flapdoodle.embed.process.store.Downloader.download(Downloader.java:69) ~[de.flapdoodle.embed.process-2.0.1.jar:na]
....
MongoDB gets downloaded just fine, when I hit the following URL in my web browser:
https://downloads.mongodb.org/win32/mongodb-win32-i386-3.2.2.zip
This leads me to believe that probably I'm missing some configuration in my Eclipse or may be the maven project itself.
Please help me to find the right configuration.
What worked for me on a windows machine:
Download the zip file (https://downloads.mongodb.org/win32/mongodb-win32-i386-3.2.2.zip)
manually and put it (not unpack) into this folder:
C:\Users\<Username>\.embedmongo\win32\
Indeed the problem is about your proxy (a corporate one I guess).
If the proxy do not require authentication, you can solve your problem easily just by adding the appropriate -Dhttp.proxyHost=... and -Dhttp.proxyPort=... (or/and the same with "https.[...]") as JVM arguments in your eclipse junit Runner, as suggested here : https://github.com/learning-spring-boot/learning-spring-boot-2nd-edition-code/issues/2
One solution to your problem is to do the following.
Download MongoDB and place it on a ftp server which is inside your corporate network (for which you would not need proxy).
Then write a configuration in your project like this
#Bean
#ConditionalOnProperty("mongo.proxy")
public IRuntimeConfig embeddedMongoRuntimeConfig() {
final Command command = Command.MongoD;
final IRuntimeConfig runtimeConfig = new RuntimeConfigBuilder()
.defaults(command)
.artifactStore(new ExtractedArtifactStoreBuilder()
.defaults(command)
.download(new DownloadConfigBuilder()
.defaultsForCommand(command)
.downloadPath("your-ftp-path")
.build())
.build())
.build();
return runtimeConfig;
}
With the property mongo.proxy you can control whether Spring Boot downloads MongoDB from your ftp server or from outside. If it is set to true then it downloads from the ftp server. If not then it tries to download from the internet.
The easiest way seems to me to customize the default configuration:
#Bean
DownloadConfigBuilderCustomizer mongoProxyCustomizer() {
return configBuilder -> {
configBuilder.proxyFactory(new HttpProxyFactory(host, port));
};
}
Got the same issue (with Spring Boot 2.6.1 the spring.mongodb.embedded.version property is mandatory).
To configure the proxy, I've added the configuration bean by myself:
#Value("${spring.mongodb.embedded.proxy.domain}")
private String proxyDomain;
#Value("${spring.mongodb.embedded.proxy.port}")
private Integer proxyPort;
#Bean
RuntimeConfig embeddedMongoRuntimeConfig(ObjectProvider<DownloadConfigBuilderCustomizer> downloadConfigBuilderCustomizers) {
Logger logger = LoggerFactory.getLogger(this.getClass().getPackage().getName() + ".EmbeddedMongo");
ProcessOutput processOutput = new ProcessOutput(Processors.logTo(logger, Slf4jLevel.INFO), Processors.logTo(logger, Slf4jLevel.ERROR), Processors.named("[console>]", Processors.logTo(logger, Slf4jLevel.DEBUG)));
return Defaults.runtimeConfigFor(Command.MongoD, logger).processOutput(processOutput).artifactStore(this.getArtifactStore(logger, downloadConfigBuilderCustomizers.orderedStream())).isDaemonProcess(false).build();
}
private ExtractedArtifactStore getArtifactStore(Logger logger, Stream<DownloadConfigBuilderCustomizer> downloadConfigBuilderCustomizers) {
de.flapdoodle.embed.process.config.store.ImmutableDownloadConfig.Builder downloadConfigBuilder = Defaults.downloadConfigFor(Command.MongoD);
downloadConfigBuilder.progressListener(new Slf4jProgressListener(logger));
downloadConfigBuilderCustomizers.forEach((customizer) -> {
customizer.customize(downloadConfigBuilder);
});
DownloadConfig downloadConfig = downloadConfigBuilder
.proxyFactory(new HttpProxyFactory(proxyDomain, proxyPort)) // <--- HERE
.build();
return Defaults.extractedArtifactStoreFor(Command.MongoD).withDownloadConfig(downloadConfig);
}
In my case, I had to add the HTTPS corporate proxy to Intellij Run Configuration.
Https because it was trying to download:
https://downloads.mongodb.org/win32/mongodb-win32-x86_64-4.0.2.zip
application.properties:
spring.data.mongodb.database=test
spring.data.mongodb.port=27017
spring.mongodb.embedded.version=4.0.2
Please keep in mind this is a (DEV) setup.

Passing service instance ID into guest executable at runtime

Is it possible to pass service instance ID (int value) into guest executable at runtime? I have looked at <ExeHost><Arguments>, but it's only good for static data that has to be provided up front.
The environment variable provides the service package instance ID which is not the same as the instance/replica ID. Generally speaking, SF's environment variables can provide the same information available using FabricRuntime, i.e. the node context and the code package activation context. In native SF services, the instance ID is provided at run-time by the Fabric (in the ServiceContext class), as a single process can host multiple partitions and instances/replicas.
In a guest executable, which does not use SF APIs, the only option AFAIK is to query the Fabric for this information in a separate executable, run it as the SetupEntryPoint (which runs every time before the guest executable) and write the information to a file.
For example (compile the code into GetFabricData.exe and add it to the code package):
private static async Task MainAsync(string[] args)
{
var serviceTypeName = args.FirstOrDefault();
if (string.IsNullOrEmpty(serviceTypeName)) throw new ArgumentNullException(nameof(serviceTypeName));
using (var client = new FabricClient())
{
var activationContext = FabricRuntime.GetActivationContext();
var nodeContext = FabricRuntime.GetNodeContext();
var nodeName = nodeContext.NodeName;
var applicationName = new Uri(activationContext.ApplicationName);
var replicas = await client.QueryManager.GetDeployedReplicaListAsync(nodeName, applicationName);
// usually taking the first may not be correct
// but in a guest executable it's unlikely there would be multiple partitions/instances
var instance = replicas.OfType<DeployedStatelessServiceInstance>()
.FirstOrDefault(c => c.ServiceTypeName == serviceTypeName);
if (instance == null)
{
throw new InvalidOperationException($"Unable to find a service instance for {serviceTypeName}");
}
File.WriteAllText("FabricData", instance.InstanceId.ToString());
}
}
And in the service manifest:
<SetupEntryPoint>
<ExeHost>
<Program>GetFabricData.exe</Program>
<Arguments>Guest1Type</Arguments>
</ExeHost>
</SetupEntryPoint>
Then, the guest executable can simply read the FabricData file.
It's available in an environment variable. See here for a complete list of environment variables available to services: https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-manage-multiple-environment-app-configuration
Note that the one you're asking for, Fabric_ServicePackageInstanceId is generally only meant for internal consumption, and it identifies the entire service package. That means if you have multiple code packages (executables) in your service package, they will all get the same ID.