downloading lastfinished build from teamcity - perl

I'm using Perl's File::Fetch to download a file from the lastfinished build in Teamcity. This is working fine except the file is versioned, but I'm not getting the version number.
sub GetTeamcityFiles {
my $latest_version = "C:/dowloads"
my $uri = "http://<teamcity>/guestAuth/repository/download/bt11/.lastFinished/MyApp.{build.number}.zip";
# fetch the uri to extract directory
my $ff = File::Fetch->new(uri => "$uri");
my $where = $ff->fetch( to => "$latest_version" );
This gives me a file:
C:\downloads\MyApp.{build.number}.zip.
However, the name of the file downloaded has a build number in the name. Unfortunately there is no version file within the zip, so this is the only way I have of telling what file i've downloaded. Is there any way to get this build number?
c:\downloads\MyApp.12345.zip

With build configs modification
If you have the ability to modify the build configs in TeamCity, you can easily embed the build number into the zip file.
Create a new build step - choose command line
For the script, do something like: echo %build.number% > version.txt
That will put version.txt at the root directory of your build folder in TeamCity, which you can include in your zip later when you create it.
You can later read that file in.
I'm not able to access my servers right now so I don't have the exact name of the parameter, but typing %build will pull up a list of TeamCity parameters to choose from, and I think it is %build.number% that you're after.
Without build configs modification
If you're not able to modify the configs, you're going to need something like egrep:
$ echo MyApp.12.3.4.zip | egrep -o '([0-9]+.){2}[0-9]+'
> 12.3.4
$ echo MyApp.1234.zip | egrep -o '[0-9]+'
> 1234
It looks like you're running on Windows; in those cases I use UnxUtils & UnxUpdates to get access to utilities like this. They're very lightweight and do not install to the registry, just add them to your system PATH.

Related

Stale file is located outside of the allowed root path when using Cuckoo

I wanted to mock some of my files, so I used Cuckoo framework. I am using Swift Package Manager, so I did every step that is shown in README of framework.
I tried to use this script
# Define output file. Change "${PROJECT_DIR}/${PROJECT_NAME}Tests" to your test's
root source folder, if it's not the default name.
OUTPUT_FILE="${PROJECT_DIR}/${PROJECT_NAME}Tests/GeneratedMocks.swift"
echo "Generated Mocks File = ${OUTPUT_FILE}"
# Define input directory. Change "${PROJECT_DIR}/${PROJECT_NAME}" to your project's root source folder, if it's not the default name.
INPUT_DIR="${PROJECT_DIR}/${PROJECT_NAME}"
echo "Mocks Input Directory = ${INPUT_DIR}"
# Generate mock files, include as many input files as you'd like to create mocks for.
"${PROJECT_DIR}/run" --download generate --testable "${PROJECT_NAME}" \
--output "${OUTPUT_FILE}" \
"${INPUT_DIR}/Common/Repository/LatestNewsRepository/LatestNewsRepositoryImpl.swift" \
# ... and so forth, the last line should never end with a backslash
# After running once, locate `GeneratedMocks.swift` and drag it into your Xcode test target group.
I also downloaded the latest run script and I had to check For install builds only.
When app is launched I am getting this error -
Stale file '.../LibraryTests/GeneratedMocks.swift' is located outside of the allowed root paths.
Things I tried -
Clean Xcode derived data
Clean build folder
Reset Xcode
Reset Packages Cache
and I am still not getting output file. Is there anything else I should try?

How to deploy releases automatically to gitlab using ci

Im currently trying to figure out how to deploy an gitlab project automatically using ci. I managed to run the building stage successfully, but im unsure how to retrieve and push those builds to the releases.
As far as I know it is possibile to use rsync or webhooks (for example Git-Auto-Deploy) to get the build. However I failed to apply these options successfully.
For publishing releases I did read https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/api/tags.md#create-a-new-release, but im not sure if I understand the required pathing schema correctly.
Is there any simple complete example to try out this process?
A way is indeed to use webhooks:
There are tons of different possible solutions to do that. I'd go with a sh script which is invoked by the hook.
How to intercept your webhook is up to the configuration of your server, if you have php-fpm installed you can use a PHP script.
When you create a webhook in your Gitlab project (Settings->Webhooks) you can specify for which kind of events you want the hook (in our case, a new build), and a secret token so you can verify the script has been called by Gitlab.
The PHP script can be something like that:
<?php
// Check token
$security_file = parse_ini_file("../token.ini");
$gitlab_token = $_SERVER["HTTP_X_GITLAB_TOKEN"];
if ($gitlab_token !== $security_file["token"]) {
echo "error 403";
exit(0);
}
// Get data
$json = file_get_contents('php://input');
$data = json_decode($json, true);
// We want only success build on master
if ($data["ref"] !== "master" ||
$data["build_stage"] !== "deploy" ||
$data["build_status"] !== "success") {
exit(0);
}
// Execute the deploy script:
shell_exec("/usr/share/nginx/html/deploy.sh 2>&1");
I created a token.ini file outside the webroot, which is just one line:
token = supersecrettoken
In this way the endpoint can be called only by Gitlab itself. The script then checks some parameters of the build, and if everything is ok it runs the deploy script.
Also the deploy script is very very basic, but there are a couple of interesting things:
#!/bin/bash
# See 'Authentication' section here: http://docs.gitlab.com/ce/api/
SECRET_TOKEN=$PERSONAL_TOKEN
# The path where to put the static files
DEST="/usr/share/nginx/html/"
# The path to use as temporary working directory
TMP="/tmp/"
# Where to save the downloaded file
DOWNLOAD_FILE="site.zip";
cd $TMP;
wget --header="PRIVATE-TOKEN: $SECRET_TOKEN" "https://gitlab.com/api/v3/projects/774560/builds/artifacts/master/download?job=deploy_site" -O $DOWNLOAD_FILE;
ls;
unzip $DOWNLOAD_FILE;
# Whatever, do not do this in a real environment without any other check
rm -rf $DEST;
cp -r _site/ $DEST;
rm -rf _site/;
rm $DOWNLOAD_FILE;
First of all, the script has to be executable (chown +x deploy.sh) and it has to belong to the webserver’s user (usually www-data).
The script needs to have an access token (which you can create here) to access the data. I inserted it as environment variable:
sudo vi /etc/environment
in the file you have to add something like:
PERSONAL_TOKEN="supersecrettoken"
and then remember to reload the file:
source /etc/environment
You can check everything is alright doing sudo -u www-data echo PERSONAL_TOKEN and verify the token is printed in the terminal.
Now, the other interesting part of the script is where is the artifact. The last available build of a branch is reachable only through API; they are working on implementing the API in the web interface so you can always download the last version from the web.
The url of the API is
https://gitlab.example.com/api/v3/projects/projectid/builds/artifacts/branchname/download?job=jobname
While you can imagine what branchname and jobname are, the projectid is a bit more tricky to find.
It is included in the body of the webhook as projectid, but if you do not want to intercept the hook, you can go to the settings of your project, section Triggers, and there are examples of APIs calls: you can determine the project id from there.

Is it possible to use relative paths in sphinx.conf?

I'm using Sphinx on a Linux production server as well as a Windows dev machine running WampServer.
The index configurations in sphinx.conf each require a path setting for the output file name. Because the filesystems on the production server and dev machine are different, I have to have two lines and then comment one out depending on which server I'm using.
#path = /path/to/folder/name #LIVE
path = C:\wamp\www\site\path\to\folder\name #LOCALHOST
Since I have lots of indexes, it gets really old having to constantly comment and uncomment dozens of lines every time I need to update the file.
Using relative paths would be the ideal solution, but when I tried that I received the following error when running the indexer:
FATAL: failed to open ../folder/name.tmp.spl: Invalid argument, will not index. Try --rotate option.
Is it possible to use relative paths in sphinx.conf?
You can use relative paths, but its kind of tricky because you the various utilities will have different working directories.
eg On windows the searchd service, will start IIRC with a working directory of $WINDIR$\System32
on linux, via crontab, I think it has working directory left over from previously, so would have to change the folder in the actual command line
... ie its not relative to the config file, its relative to the current working directory.
Personally I use a version control system (SVN actually) to manage it. The version from Dev, is always the one commited to the repository, the 'working copy' on the LIVE server, has had the paths edited to the right location. So when 'update' to the latest file, only changes are merged leaving the local filepaths in tact.
Other people use a dynamic config file. The config file can be a script (php/python/perl etc) - but this only works on linux so wont help you.
Or can just have a 'publish' script. Basically, you edit a 'master' config file, and one that can be freely copied to all servers. Then a 'publish' script, that writes the apprirate local path. It could do it with some pretty simple search replace.
<?php
if (trim(`hostname`) == 'live') {
$path = '/path/to/folder/';
} else {
$path = 'C:\wamp\www\site\path\to\folder\`;
}
$contents = file_get_contents('sphinx.conf.master');
$contents = str_replace('$path',$path,$contents);
file_put_contents('sphinx.conf',$contents);
Then have path = $path\name in the master config file, which will get replaced to the proper path, when run the script on the local machine

Extract metadata for package component files using Tika

I am trying to extract metadata for package component files using Tika at the command line, but I can only seem to get it to output metadata for the containing package file. Example: test_file.zip contains two files, test1.doc and test2.doc. I want to get the metadata for test1.doc and test2.doc, but cannot figure out how to do so.
I tried to run this:
java -jar tika-app-1.5.jar -m test_files.zip
but that just outputted the Content-Length, Content-Type, and resourceName for test_files.zip.
I also tried to run this:
java -jar tika-app-1.5.jar -h test_files.zip
That outputted the HTML for each component file, wrapped in a <div> with class ."package-entry", but the metadata tags were again outputted only for the containing package file test_files.zip. I tried using the -x parameter instead of -h, and no parameter at all, and got exactly the same result.
How do I get the metadata for the component files? I don't mind parsing the embedded metadata from xhtml but I cannot figure how to get it injected into the xhtml or otherwise outputted.
Any help much appreciated. Thank you.
Since you've said you want to do it with only the tika-app jar, your best option is something like
# Create a temp directory
cd /tmp
mkdir tika-extracted
cd tika-extracted
# Have Tika extract out all the embedded resources
java -jar tika-app-1.5.jar --extract $INPUT
# Process each one in turn
for e in *; do
java -jar tika-app-1.5.jar --metadata $e
done
# Tidy up
cd /tmp
rm -rf tika-extracted
Using Java, you'd be able to register your own EmbeddedDocumentExtractor on the ParserContext, and use that to trigger the metadata extraction for each one individually

Removing files with Build Phase?

Is it possible to remove a file using a build phase in xcode 4 based on if it is release or dev?
If so has anyone got an example?
I have tried :
if [ "${CONFIGURATION}" = "Debug" ]; then
find "$TARGET_BUILD_DIR" -name '*-live.*' -print0 | xargs -0 rm
fi
This prints CopyStringsFile
"build/Debug-iphonesimulator/Blue Sky.app/PortalText-live.strings" CDL/PortalText-live.strings
cd "/Users/internet/Desktop/iPhone Template/iPhonePortalTemplate/CDL.Labs"
setenv PATH "/Developer/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin:/Developer/Applications/Xcode.app/Contents/Developer/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin"
builtin-copyStrings --validate --inputencoding utf-8 --outputencoding binary --outdir "/Users/internet/Desktop/iPhone Template/iPhonePortalTemplate/CDL.Labs/build/Debug-iphonesimulator/Blue Sky.app" -- CDL/PortalText-live.strings
But does actually remove the file from the bundle.
The only way I've ever had different files, is having a separate Target, and only include certain files in certain targets.
EDIT WITH AN EXAMPLE
Ok, I've done exactly the same in another project. We had a DefaultProperties.plist file, which was included in the target.
We then had 3 copies of this, NOT included in the target, ProdProperties.plist, TestProperties.plist, UatProperties.plist.
We built for environments on the command line, using xcodebuild, as it was built using an automated build server (Bamboo).
Prior to executing xcodebuild, we would run this:
cp -vf "./Properties/Environments/${environment}Properties.plist" ./Properties/shared/DefaultProperties.plist
touch Properties/shared/DefaultProperties.plist
with $(environment) being passed into the script.
You could do something like this with the RunScript phase in Xcode.