I have a use case wherein, I need to connect to an API for data request.
The API requires a valid token to process the requests.
To generate the token, I have a accountCode & secret key
Assume BaseURL as
BaseURL - http://api.xxxxx.com/{accountCode}/data (Value of account needs to be passed)
**Below script in Python/Java needs to be run to fetch the dateToken & token
If we use Python 3.6 or above. Below is the code -**
--START-- {
import time
import requests
from hashlib import md5
account_code = "<account_name>"
key = "<api_key>"
actual_unix_time = int(time.time_ns() / 1000) # in milliseconds
TTL = 31536000000 # for 1 year
expiration_time = actual_unix_time + TTL
base_url = "https://api.xxxxx.com"
url = f"/{account_code}/data?fromDate=last6Hours&granularity=minute&type=ALL%2CVOD%2CLIVE&operation=reduceJoin&metrics=bufferratio"
pre_url = f"{url}&dateToken={expiration_time}"
token_generated = md5(f"{pre_url}{key}".encode('utf-8'))
token_value = token_generated.hexdigest()
request_url = f"{base_url}{pre_url}&token={token_value}"
response = requests.get(request_url)
print(response)
print(response.text)
} --END--
- If we use Java. Below is the code -
--START-- {
var key = pm.environment.get("NPAW-API-KEY");
var base_url = "https://api.xxxxx.com";
var url = pm.request.url.toString();
var path = url.replace(base_url, '');
var pre_url = pm.variables.replaceIn(path);
var moment = require('moment');
var actual_unix_time = moment().unix()*1000;
var TTL = 31536000000
var expiration_time = (actual_unix_time + TTL);
var pre_url = pre_url+"&dateToken="+expiration_time;
var token_generated = CryptoJS.MD5(pre_url + key).toString();
var token_value = token_generated;
var request_url = (base_url+pre_url+'&token='+token_value).toString();
}--END--
Example of how the final URL - https://api.xxxxx.com/kb-vivek/data?fromDate=today&granularity=hour&type=ALL,VOD,LIVE&operation=reduceJoin&metrics=views,playtime&dateToken=1699016056000&token=7a9c97a4d4f108d1d32be2f7f8d00731
I tried to use Postman, wherein, I could pass the above script in the Pre-Request script and set environment variables for accountCode & Secret Key and I was able to achieve the result as desired.
Question: How can I achieve this in Azure Data Factory?
To achieve the requirement, we need to use a combination of set variables and dataflows (to generate md5 hex string and store final url in a file).
First, I have created 4 parameters with values as shown below:
base_url: https://api.xxxxx.com
account_code: <account_name>
key: <api_key>
TTL: 31536000000
First, I have created a variable to build url. I used the following dynamic content:
/#{pipeline().parameters.account_code}/data?fromDate=last6Hours&granularity=minute&type=ALL%2CVOD%2CLIVE&operation=reduceJoin&metrics=bufferratio
Next, I have built the pre_url with the following dynamic content:
#{variables('url')}&dateToken=#{add(div(sub(ticks(utcNow()), ticks('1970-01-01')),10),31536000000)}
Now, to encode the string and convert it to md5 hex string, I have used dataflow. I have passed base_url, pre_url and key to the dataflow from the pipeline.
I have taken a sample csv file with only one row from blob storage (the data in this file does not matter but make sure it has 1 row only).
I have created a derived column to create final URL by concatenating base_url, pre_url and encoded md5 hex string. Use the following content:
$base_url+$pre_url+'&token='+md5(encode(concat($pre_url,$key)))
Now I am writing this data to a file by using output to single file option in the sink settings.
When I debug the pipeline, the file will be written to my storage account. The contents of the file will be as shown below:
NOTE:
Now since you want to generate date token once and use it for a year, I have written the data to a file.
You run this pipeline once and generate a file with required URL (as above). Anytime you want to access this URL, you can use look up activity to access the URL anywhere required.
I have used utcNow() to generate dateToken. But if you have any specific date in mind, you can simply use that in the correct format (in place of utcNow()).
I've got a Scala Akka App where I execute python scripts inside Futures with ProcessBuilder.
Unfortunately are special character not displayed correct, so do I get instead of mädchen-> m�dchen
(äöü -> �)
If I execute the python script via command line do I get the right output of "mädchen", so I assume it has nothing to do with the python script instead somehow related to my Scala input read.
Python Spider:
print("mädchen")
Scala:
val proc = Process("scrapy runspider spider.py")
var output : String = ""
val exitValue = proc ! ProcessLogger (
(out) => if( out.trim.length > 0 )
output += out.trim,
(err) =>
System.err.printf("e:%s\n",err)
)
println(exitValue) // 0 -> succ.
println(output) // m�dchen -> should be mädchen
I already tried many thinks and also read that Strings are by default UTF-8 so I am not sure why I get those question marks.
Also did I tried with no success:
var byteBuffer : ByteBuffer = StandardCharsets.UTF_8.encode(output.toString())
val str = new String(output.toString().getBytes(), "UTF-8")
Update:
It seems to be a windows related issue, following instruction will solve this problem: Using UTF-8 Encoding (CHCP 65001) in Command Prompt / Windows Powershell (Windows 10)
i am trying to make a game where players create their own buildings and can then save them for other players to see and play on. However, roblox doesn't let me store all the data needed for the whole creation(there are several properties for each brick)
All i get is this error code:
104: Cannot store Array in DataStore
any help would be greatly appreciated!
I'm not sure if this is the best method, but it's my attempt. Below is an example of a table, you can use tables to store several values. I think you can use HttpService's JSONEncode function to convert tables into strings (which hopefully can be saved more efficiently)
JSONEncode (putting brick's data into a string, which you can save into the DataStore
local HttpService = game:GetService("HttpService")
-- this is an example of what we'll convert into a json string
local exampleBrick = {
["Size"] = Vector3.new(3,3,3),
["Position"] = Vector3.new(0,1.5,0),
["BrickColor"] = BrickColor.new("White")
["Material"] = "Concrete"
}
local brickJSON = HttpService:JSONEncode(exampleBrick)
print(brickJSON)
-- when printed, you'll get something like
-- { "Size": Vector3.new(3,3,3), "Position": Vector3.new(0,1.5,0), "BrickColor": BrickColor.new("White"), "Material": "Concrete"}
-- if you want to refer to this string in a script, surround it with two square brackets ([[) e.g. [[{"Size": Vector3.new(3,3,3)... }]]
JSONDecode (reading the string and converting it back into a brick)
local HttpService = game:GetService("HttpService")
local brickJSON = [[ {"Size": Vector3.new(3,3,3), "Position": Vector3.new(0,1.5,0), "BrickColor": BrickColor.new("White"), "Material": "Concrete"} ]]
function createBrick(tab)
local brick = Instance.new("Part")
brick.Parent = <insert parent here>
brick.Size = tab[1]
brick.Position= tab[2]
brick.BrickColor= tab[3]
brick.Material= tab[4]
end
local brickData = HttpService:JSONDecode(brickJSON)
createBrick(brickData) --this line actually spawns the brick
The function can also be wrapped in a pcall if you want to account for any possible datastore errors.
Encoding a whole model into a string
Say your player's 'building' is a model, you can use the above encode script to convert all parts inside a model into a json string to save.
local HttpService = game:GetService("HttpService")
local StuffWeWantToSave = {}
function getPartData(part)
return( {part.Size,part.Position,part.BrickColor,part.Material} )
end
local model = workspace.Building --change this to what the model is
local modelTable = model:Descendants()
for i,v in pairs(modelTable) do
if v:IsA("Part") or v:IsA("WedgePart") then
table.insert(StuffWeWantToSave, HttpService:JSONEncode(getPartData(modelTable[v])))
end
end
Decoding a string into a whole model
This will probably occur when the server is loading a player's data.
local HttpService = game:GetService("HttpService")
local SavedStuff = game:GetService("DataStoreService"):GetDataStore("blabla") --I don't know how you save your data, so you'll need to adjust this and the rest of the scripts (as long as you've saved the string somewhere in the player's DataStore)
function createBrick(tab)
local brick = Instance.new("Part")
brick.Parent = <insert parent here>
brick.Size = tab[1]
brick.Position= tab[2]
brick.BrickColor= tab[3]
brick.Material= tab[4]
end
local model = Instance.new("Model") --if you already have 'bases' for the players to load their stuff in, remove this instance.new
model.Parent = workspace
for i,v in pairs(SavedStuff) do
if v[1] ~= nil then
CreateBrick(v)
end
end
FilteringEnabled
If your game uses filteringenabled, make sure that only the server handles saving and loading data!! (you probably already knew that) If you want the player to save by clicking a gui button, make the gui button fire a RemoteFunction that sends their base's data to the server to convert it to a string.
BTW I'm not that good at scripting so I've probably made a mistake somehwere.. good luck though
Crabway's answer is correct in that the HttpService's JSONEncode and JSONDecode methods are the way to go about tackling this problem. As it says on the developer reference page for the DataStoreService, Data is ... saved as a string in data stores, regardless of its initial type. (https://developer.roblox.com/articles/Datastore-Errors.) This explains the error you received, as you cannot simply push a table to the data store; instead, you must first encode a table's data into a string using JSONEncode.
While I agree with much of Crabway's answer, I believe the function createBrick would not behave as intended. Consider the following trivial example:
httpService = game:GetService("HttpService")
t = {
hello = 1,
goodbye = 2
}
s = httpService:JSONEncode(t)
print(s)
> {"goodbye":2,"hello":1}
u = httpService:JSONDecode(s)
for k, v in pairs(u) do print(k, v) end
> hello 1
> goodbye 2
As you can see, the table returned by JSONDecode, like the original, uses strings as keys rather than numeric indices. Therefore, createBrick should be written something like this:
function createBrick(t)
local brick = Instance.new("Part")
brick.Size = t.Size
brick.Position = t.Position
brick.BrickColor = t.BrickColor
brick.Material = t.Material
-- FIXME: set any other necessary properties.
-- NOTE: try to set parent last for optimization reasons.
brick.Parent = t.Parent
return brick
end
As for encoding a model, calling GetChildren would produce a table of the model's children, which you could then loop through and encode the properties of everything within. Note that in Crabway's answer, he only accounts for Parts and WedgeParts. You should account for all parts using object:IsA("BasePart") and also check for unions with object:IsA("UnionOperation"). The following is a very basic example in which I do not store the encoded data; rather, I am just trying to show how to check the necessary cases.
function encodeModel(model)
local children = model:GetChildren()
for _, child in ipairs(children) do
if ((child:IsA("BasePart")) or (child:IsA("UnionOperation"))) then
-- FIXME: encode child
else if (child:IsA("Model")) then
-- FIXME: using recursion, loop through the sub-model's children.
end
end
return
end
For userdata, such as Vector3s or BrickColors, you will probably want to convert those to strings when you go to encode them with JSONEncode.
-- Example: part with "Brick red" BrickColor.
color = tostring(part.BrickColor)
print(string.format("%q", color))
> "Bright red"
I suggest what #Crabway said, use HttpService.
local httpService = game:GetService("HttpService")
print(httpService:JSONEncode({a = "b", b = "c"}) -- {"a":"b","b":"c"}
But if you have any UserData values such as Vector3s, CFrames, Color3s, BrickColors and Enum items, then use this library by Defaultio. It's actually pretty nice.
local library = require(workspace:WaitForChild("JSONWithUserdata"))
library:Encode({Vector3.new(0, 0, 0)})
If you want a little documentation, then look at the first comment in the script:
-- Defaultio
--[[
This module adds support for encoding userdata values to JSON strings.
It also supports lists which skip indices, such as {[1] = "a", [2] = "b", [4] = "c"}
Userdata support is implemented by replacing userdata types with a new table, with keys _T and _V:
_T = userdata type enum (index in the supportedUserdataTypes list)
_V = a value or table representing the value
Follow the examples bellow to add suppport for additional userdata types.
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
Usage example:
local myTable = {CFrame.new(), BrickColor.Random(), 4, "String", Enum.Material.CorrodedMetal}
local jsonModule = require(PATH_TO_MODULE)
local jsonString = jsonModule:Encode(myTable)
local decodedTable = jsonModule:Decode(jsonString)
--]]
I have:
Raw xml filled by a select query.This xml transformed into a HL7
message
One of the tags of this xml represents a clob column from a table in
the database
I mapped this data (from edit transformer section) as a variable.
Now I am trying to convert this variable into a base64 string then
replace it in my transformed hl7 message.
5.I tried this conversion on a destination channel which is a javascript writer.
I read and tried several conversion methods like
Packages.org.apache.commons.codec.binary.Base64.encodeBase64String();
I have got only error messages like :
EvaluatorException: Can't find method org.apache.commons.codec.binary.Base64.encodeBase64String(java.lang.String);
Code piece:
var ads=$('V_REPORT_CLOB');
var encoded = Packages.org.apache.commons.codec.binary.Base64.encodeBase64String(ads.toString());
It is pretty clear that I am a newbie on that.How can I manage to do this conversion ?
Here is what I use for Base64 encoding a string with your var substituted.
//Encode Base 64//
var ads = $('V_REPORT_CLOB');
var adsLength = ads.length;
var base64Bytes = [];
for(i = 0; i < adsLength;i++){
base64Bytes.push(ads.charCodeAt(i));
}
var encodedData = FileUtil.encode(base64Bytes);
I'm trying to store user-uploaded images in my application which is written by scala and play framework 2.2.x
I've deployed my app in heroku.
Heroku does not allow me to save my file in file system.
So I've tried to store my file in data base.
here is the code that I use for storing image :
def updateImage(id: Long, image: Array[Byte]) = {
val selected = getById(id)
DB.withConnection {
implicit c =>
SQL("update subcategory set image={image} where id = {id}").on('id -> id, 'image -> image).executeUpdate()
}
selected }
and here is the code that I use to retreive my image :
def getImageById(id: Long): Array[Byte] = DB.withConnection {
implicit c =>
val all = SQL("select image from subcategory where id = {id}").on('id -> id)().map {
case Row(image: Array[Byte]) => image
case Row(Some(image: Array[Byte])) => image
case Row(image: java.sql.Blob )=> image.getBytes(0 , image.length().toInt)
}
all.head
}
The problem is: when I use H2 database and blob column, I get the "Match Error" exception.
When I use Postgresql and bytea column, I got no error but when I retrieve the image, It's in hex format and some of the bytes in the beginning of the array are missing.
According to the PostgreSQL documentation, bytea stores the length of the array in the four bytes at the beginning of the array. These are stripped when you read the row, so that's why they seem to be "missing" when you compare the data in Scala with the data in the DB.
You will have to set the response's content-type to the appropriate value if you want the web browser to display the image correctly, as otherwise it does not know it is receiving image data. The Ok.sendFile helper does it for you. Otherwise you will have to do it by hand:
def getPicture = Action {
SimpleResult(
header = ResponseHeader(200),
body = Enumerator(pictureByteArray))
.as(pictureContentType)
}
In the example above, pictureByteArray is the Array[Byte] containing the picture data from your database, and pictureContentType is a string with the appropriate content type (for example, image/jpeg).
This is all quite well explained in the Play documentation.