Functions to deal with Unicode string manipulation with PowerBuilder 12.1 - unicode

Question

I am currently working on converting our PowerBuilder 12.1 application, which does not currently support Unicode, into a Unicode supporting application.
I have made some modifications to save Unicode data to our database, as well to files, but I have hit a slight snag in processing strings.
For example, the character 𠆾 is a Surrogate Pair and PowerBuilder interprets this as 2 characters (similar to how .NET operates). Thus:
LEN("𠆾") = 2
To me, this part makes sense, as it is count each code unit as a character.
Currently we have come up with two solutions to handle doing string functions with Unicode characters:
Callable OLEObjects written in C# .NET
using the PBNI interface to call C# .NET (want to stay away from this solution if possible)
An example of the .NET code we are thinking of using for determining the string length is:
StringInfo.ParseCombiningCharacters("𠆾").Length = 1
We are just worried about the impact on performance with constantly calling the OLEObjects/PBNI to do all of our string processing. Have any of the other PowerBuilder developers here done Unicode string manipulation (LEN, MID, POS, etc), and how did you do it?
Thank you.

Answer 1

This is in response to Seki's hex conversion function. I'm posting it as an answer so I can include source code. I use the Microsoft cryptographic functions to display blobs in my debugging tools. Here's a simplified version of my blob window. The one I use is PFC-based and uses an object that wraps the MS Crypto library. It's from PB 12.5 but should import into any Unicode version of PB.
HA$PBExportHeader$w_show_blob.srw
forward
global type w_show_blob from window
end type
type sle_1 from singlelineedit within w_show_blob
end type
type mle_1 from multilineedit within w_show_blob
end type
end forward
global type w_show_blob from window
integer width = 3081
integer height = 1988
boolean titlebar = true
boolean controlmenu = true
boolean minbox = true
boolean maxbox = true
boolean resizable = true
boolean center = true
sle_1 sle_1
mle_1 mle_1
end type
global w_show_blob w_show_blob
type prototypes
FUNCTION boolean CryptBinaryToString ( &
Blob pbBinary, &
ulong cbBinary, &
ulong dwFlags, &
Ref string pszString, &
Ref ulong pcchString ) &
LIBRARY "crypt32.dll" ALIAS FOR "CryptBinaryToStringW"
end prototypes
type variables
CONSTANT Ulong CRYPT_STRING_HEXASCIIADDR = 11
end variables
forward prototypes
public subroutine of_showblob (ref blob abl_data)
end prototypes
public subroutine of_showblob (ref blob abl_data);unsignedlong lul_size, lul_bufsize
string ls_hex
try
lul_size = len(abl_data)
sle_1.text = string(lul_size)
setnull(ls_hex)
cryptbinarytostring( abl_data, lul_size, CRYPT_STRING_HEXASCIIADDR, ls_hex, lul_bufsize)
ls_hex = space(lul_bufsize)
if not cryptbinarytostring( abl_data, lul_size, CRYPT_STRING_HEXASCIIADDR , ls_hex, lul_bufsize) then
mle_1.text = "error converting blob data"
else
mle_1.text = ls_hex
end if
catch(runtimeerror re)
messagebox("oops", re.text)
end try
end subroutine
on w_show_blob.create
this.sle_1=create sle_1
this.mle_1=create mle_1
this.Control[]={this.sle_1,&
this.mle_1}
end on
on w_show_blob.destroy
destroy(this.sle_1)
destroy(this.mle_1)
end on
type sle_1 from singlelineedit within w_show_blob
integer x = 73
integer width = 517
integer height = 88
integer taborder = 10
integer textsize = -10
integer weight = 400
fontcharset fontcharset = ansi!
fontpitch fontpitch = variable!
fontfamily fontfamily = swiss!
string facename = "Arial"
long textcolor = 33554432
long backcolor = 553648127
string text = "none"
boolean displayonly = true
borderstyle borderstyle = stylelowered!
end type
type mle_1 from multilineedit within w_show_blob
integer x = 64
integer y = 96
integer width = 2898
integer height = 1716
integer taborder = 10
integer textsize = -10
integer weight = 400
fontcharset fontcharset = ansi!
fontpitch fontpitch = fixed!
fontfamily fontfamily = modern!
string facename = "Courier New"
long textcolor = 33554432
string text = "none"
boolean hscrollbar = true
boolean vscrollbar = true
boolean displayonly = true
borderstyle borderstyle = stylelowered!
end type
To use it, assuming your blob is lbl_myBlob:
open(w_show_blob)
w_show_blob.of_showblob(lbl_myBlob)
The output in the MLE looks like this:
0000 42 4d ee 00 00 00 00 00 00 00 76 00 00 00 28 00 BM........v...(.
0010 00 00 10 00 00 00 0f 00 00 00 01 00 04 00 00 00 ................
0020 00 00 78 00 00 00 00 00 00 00 00 00 00 00 00 00 ..x.............
0030 00 00 00 00 00 00 00 00 00 00 00 00 80 00 00 80 ................
0040 00 00 00 80 80 00 80 00 00 00 80 00 80 00 80 80 ................
0050 00 00 80 80 80 00 c0 c0 c0 00 00 00 ff 00 00 ff ................
0060 00 00 00 ff ff 00 ff 00 00 00 ff 00 ff 00 ff ff ................
0070 00 00 ff ff ff 00 88 88 88 88 88 88 88 88 88 88 ................
0080 80 88 88 88 88 88 88 88 80 08 88 88 88 88 88 88 ................
0090 80 00 88 88 88 88 88 88 80 00 08 88 88 88 88 88 ................
00a0 80 00 00 88 88 88 88 88 80 00 00 08 88 88 88 88 ................
00b0 80 00 00 00 88 88 88 88 80 00 00 08 88 88 88 88 ................
00c0 80 00 00 88 88 88 88 88 80 00 08 88 88 88 88 88 ................
00d0 80 00 88 88 88 88 88 88 80 08 88 88 88 88 88 88 ................
00e0 80 88 88 88 88 88 88 88 88 88 88 88 88 88 ..............

Answer 2

Since the release 10, PB is unicode (utf-16le)-aware. So the legacy Len() is implicit LenW() (as other string functions, and dealing with legacy data could imply to use explicit LenA()).
Are you sure that you are getting some utf-16le encoding ? Given the following function, what does it return on a string containing your data, if you call it with hexdump_blob(blob(your_string))?
Paste this code into the source code of a new global function named hexdump_blob to have an hexadecimal display (hex editor like) for blob contents.
global type hexdump_blob from function_object
end type
forward prototypes
global function string hexdump_blob (blob abl_data, boolean ab_fill_lastline)
end prototypes
global function string hexdump_blob (blob abl_data, boolean ab_fill_lastline);//hexify a blob content
string ls_tohex = "0123456789ABCDEF"
string ls_msg = "", ls_line, ls_binary
long i, j, length
byte b
string ls_fill
if isnull( abl_data ) then return ""
if ab_fill_lastline then
ls_fill = " __"
else
ls_fill = " "
end if
length = len( abl_data )
for i = 1 to length
GetByte( abl_data, i, b )
ls_line += mid( ls_tohex, 1+ mod(int(b/16),16), 1)
ls_line += mid( ls_tohex, 1+ mod(b,16), 1)
ls_line += " "
ls_binary += string( iif(b>31 and b<128,char(b)," "))
if mod(i,16) = 0 and i > 0 then
ls_binary = replaceall( ls_binary, "~r", "·") //no cr/lf
ls_binary = replaceall( ls_binary, "~n", "·")
ls_binary = replaceall( ls_binary, "~t", "·")
ls_msg += "[" + string( i - 16, "0000") + "] " + ls_line + "~t" + ls_binary + "~r~n"
ls_line = ""
ls_binary = ""
end if
next
i -- // i - 1 due to the last loop in for
ls_line += fill(ls_fill, 3 * ( 16 - mod(i, 16) ) )
ls_msg += "[" + string( i - mod(i,16), "0000") + "] " + ls_line + "~t" + ls_binary
return ls_msg
end function
Also, here is the replaceall() function that is used by hexdump_blob()
global type replaceall from function_object
end type
forward prototypes
global function string replaceall (string as_source, string as_pattern, string as_replace)
end prototypes
global function string replaceall (string as_source, string as_pattern, string as_replace);//remplace toute les occurences de as_pattern de as_source par as_replace
string ls_target
long i, j
ls_target=""
i = 1
j = 1
do
i = pos( as_source, as_pattern, j )
if i>0 then
ls_target += mid( as_source, j, i - j )
ls_target += as_replace
j = i + len( as_pattern )
else
ls_target += mid( as_source, j )
end if
loop while i>0
return ls_target
end function
and the iif() that simulates the C ternary operator, or the visual basic iif()
global type iif from function_object
end type
forward prototypes
global function any iif (boolean ab_cond, any aa_true, any aa_false)
end prototypes
global function any iif (boolean ab_cond, any aa_true, any aa_false);
// simulates the VB iif or C ternary operator
if ab_cond then
return aa_true
else
return aa_false
end if
end function

Answer 3

Wouldn't you want to use the LenA() method?
http://www.techno-kitten.com/Changes_to_PowerBuilder/New_in_PowerBuilder_10/PB10New_-_Unicode_Support/PB10New_-_Unicode_Related_Chan/PB10New_-_String-Related_Funct/pb10new_-_modified_processing_.html