As using hardcoded strings in scala(with play) is considered a sin. I have some hardcoded numbers in my code. So can i declare these numbers in the Messages file of play and then access them in project & convert them to Int/BigInteger whatever?
for storing configuration data use application.conf which is located in ./conf directory and then use Play.current.configuration.getInt("some.property") to retrieve it
Related
Context
Spark reader has the function format, which is used to specify a data source type, for example, JSON, CSV or third party com.databricks.spark.redshift
Help
how can I check whether a third-party format exists or not, let me give a case
In local spark, connect to redshift two open source libs available 1. com.databricks.spark.redshift 2. io.github.spark_redshift_community.spark.redshift, how I can determine which libs the user pastes in the classpath
What I tried
Class.forName("com.databricks.spark.redshift"), not worked
I tried to check spark code for how they are throwing error, here is line, but unfortunately Utils is not available publically
Instead of targeting format option, I tried to target JAR file System.getProperty("java.class.path")
spark.read.format("..").load() in try/catch
I looking for a proper & reliable solution
May this answer help you.
To only check whether is spark format exists or not,
spark.read.format("..").load() in try/catch
is enough.
And as all data sources usually register themselves using DataSourceRegister interface (and use shortName to provide their alias):
You can use Java's ServiceLoader.load method to find all registered implementations of DataSourceRegister interface.
import java.util.ServiceLoader
import org.apache.spark.sql.sources.DataSourceRegister
val formats = ServiceLoader.load(classOf[DataSourceRegister])
import scala.collection.JavaConverters._
formats.asScala.map(_.shortName).foreach(println)
I have a set of .xml documents that I want to parse.
I previously have tried to parse them using methods that take the file contents and dump them into a single cell, however I've noticed this doesn't work in practice since I'm seeing slower and slower run times, often with one task taking tens of hours to run:
The first transform of mine takes the .xml contents and puts it into a single cell, and a second transform takes this string and uses Python's xml library to parse the string into a document. This document I'm then able to extract properties from and return a DataFrame.
I'm using a UDF to conduct the process of mapping the string contents to the fields I want.
How can I make this faster / work better with large .xml files?
For this problem, we're going to combine a couple of different techniques to make this code both testable and highly scalable.
Theory
When parsing raw files, you have a couple of options you can consider:
❌ You can write your own parser to read bytes from files and convert them into data Spark can understand.
This is highly discouraged whenever possible due to the engineering time and unscalable architecture. It doesn't take advantage of distributed compute when you do this as you must bring the entire raw file to your parsing method before you can use it. This is not an effective use of your resources.
⚠ You can use your own parser library not made for Spark, such as the XML Python library mentioned in the question
While this is less difficult to accomplish than writing your own parser, it still does not take advantage of distributed computation in Spark. It is easier to get something running, but it will eventually hit a limit of performance because it does not take advantage of low-level Spark functionality only exposed when writing a Spark library.
✅ You can use a Spark-native raw file parser
This is the preferred option in all cases as it takes advantage of low-level Spark functionality and doesn't require you to write your own code. If a low-level Spark parser exists, you should use it.
In our case, we can use the Databricks parser to great effect.
In general, you should also avoid using the .udf method as it likely is being used instead of good functionality already available in the Spark API. UDFs are not as performant as native methods and should be used only when no other option is available.
A good example of UDFs covering up hidden problems would be string manipulations of column contents; while you technically can use a UDF to do things like splitting and trimming strings, these things already exist in the Spark API and will be orders of magnitude faster than your own code.
Design
Our design is going to use the following:
Low-level Spark-optimized file parsing done via the Databricks XML Parser
Test-driven raw file parsing as explained here
Wire the Parser
First, we need to add the .jar to our spark_session available inside Transforms. Thanks to recent improvements, this argument, when configured, will allow you to use the .jar in both Preview/Test and at full build time. Previously, this would have required a full build but not so now.
We need to go to our transforms-python/build.gradle file and add 2 blocks of config:
Enable the pytest plugin
Enable the condaJars argument and declare the .jar dependency
My /transforms-python/build.gradle now looks like the following:
buildscript {
repositories {
// some other things
}
dependencies {
classpath "com.palantir.transforms.python:lang-python-gradle-plugin:${transformsLangPythonPluginVersion}"
}
}
apply plugin: 'com.palantir.transforms.lang.python'
apply plugin: 'com.palantir.transforms.lang.python-defaults'
dependencies {
condaJars "com.databricks:spark-xml_2.13:0.14.0"
}
// Apply the testing plugin
apply plugin: 'com.palantir.transforms.lang.pytest-defaults'
// ... some other awesome features you should enable
After applying this config, you'll want to restart your Code Assist session by clicking on the bottom ribbon and hitting Refresh
After refreshing Code Assist, we now have low-level functionality available to parse our .xml files, now we need to test it!
Testing the Parser
If we adopt the same style of test-driven development as here, we end up with /transforms-python/src/myproject/datasets/xml_parse_transform.py with the following contents:
from transforms.api import transform, Output, Input
from transforms.verbs.dataframes import union_many
def read_files(spark_session, paths):
parsed_dfs = []
for file_name in paths:
parsed_df = spark_session.read.format('xml').options(rowTag="tag").load(file_name)
parsed_dfs += [parsed_df]
output_df = union_many(*parsed_dfs, how="wide")
return output_df
#transform(
the_output=Output("my.awesome.output"),
the_input=Input("my.awesome.input"),
)
def my_compute_function(the_input, the_output, ctx):
session = ctx.spark_session
input_filesystem = the_input.filesystem()
hadoop_path = input_filesystem.hadoop_path
files = [hadoop_path + "/" + file_name.path for file_name in input_filesystem.ls()]
output_df = read_files(session, files)
the_output.write_dataframe(output_df)
... an example file /transforms-python/test/myproject/datasets/sample.xml with contents:
<tag>
<field1>
my_value
</field1>
</tag>
And a test file /transforms-python/test/myproject/datasets/test_xml_parse_transform.py:
from myproject.datasets import xml_parse_transform
from pkg_resources import resource_filename
def test_parse_xml(spark_session):
file_path = resource_filename(__name__, "sample.xml")
parsed_df = xml_parse_transform.read_files(spark_session, [file_path])
assert parsed_df.count() == 1
assert set(parsed_df.columns) == {"field1"}
We now have:
A distributed-compute, low-level .xml parser that is highly scalable
A test-driven setup that we can quickly iterate on to get our exact functionality right
Cheers
I have gateway.properties file and it has following content:
current.path=c:/projects/sdk/
log.path=storage/logs
Here, in this file:
Either, I want to add third variable which is current.log.path and its value must be:
current.log.path=current.path + log.path
Or, I want to append string to current.path variable as below:
current.path=current.path+log.path
I have a situation that I can't do this concatenation/append job in my Java file. In Java its pretty simple but I have no idea how to do within file itself.
There is no built-in mechanism to have something like variables in a property file. You could try Apache Commons Configuration. I think it is what you are looking for.
I have a project where I've setup localization using NSLocalizedString with keys instead of actual values for text parameters, i.e. something like this:
NSLocalizedString("RunningDistance", "distance for a marathon")
instead of this:
NSLocalizedString("Running distance.", "distance for a marathon")
Then, I exported them to xliff for translation and imported back; and it's all working.
Now, I want to add new strings (and lots of them).
I read that it is currently not recommended (and supported) to use genstrings for swift.
I thought I should export the development language to xliff, add translations and reimport. I tried this and I get the The XLIFF file does not contain a target language error.
Does this mean I need to add all those strings manually into Localizable.strings and elsewhere (as I'm using base localizations)?
Is there any other way to get this done, and import my base language xliff?
In my application, I am using a plist. Please, can anyone explain what are the uses of plist with an example or a sample code?
In the context of iPhone development, Property Lists are a key-value store that your application can use to save and retrieve persistent data.
All iPhone applications have at least one of these by default, the Information Property List:
The information property list is a
file named Info.plist that is included
with every iPhone application project
created by Xcode. It is a property
list whose key-value pairs specify
essential runtime-configuration
information for the application. The
elements of the information property
list are organized in a hierarchy in
which each node is an entity such as
an array, dictionary, string, or other
scalar type.
Plist are XML files in a specific format. Prior to XML, they had a custom format now called 'old plist'. (You almost never see that anymore save in legacy code.)
Foundations collection classes automatically generate XML files in the plist format when you use their serialization methods to write them to disk. They also automatically read them back. You can also write your own serializers for your own custom objects. This allows you to persistently store complex objects in a robust, human readable format.
One use for plist for programmers is that it is easier to use the plist editor to input and manage a lot of data than it is to try and code it. For example, if you have an class that requires setting a large number of ivars, you can create a plist, read it into an NSArray or NSDictionary and then initialize the instance by passing it the dictionary.
I use this technique when I have to use a large number of paths to draw complex objects. You define the path in the plist file instead of the code and edit the path in the plist editor.
It's also a handy way to create a large amount of detailed test data.
PList means PropertyList
It is XML file format
It is mainly user for store and reterve the data
It can store the key-value pair
It's been a long time since I've looked at them, but plist is a short-form of "properties list" and can be used to store application configuration settings that need to persist between instances of an application's execution. Could be similar to a .properties file (I see those a lot on Java projects).
A plist is essentially just a data file, it stores information in a documented format.
From Wikipedia:
In the Mac OS X Cocoa, NeXTSTEP, and
GNUstep programming frameworks,
property list files are files that
store serialized objects. Property
list files use the filename extension
.plist, and thus are often referred to
as plist files. Property list files
are often used to store a user's
settings. They are also used to store
information about bundles and
applications, a task served by the
resource fork in the old Mac OS.
.plist
Info.plist is key/value persistence storage(property list) which is used by system and user. It contains user-friendly text in XML format. Info.plist is mandatory file for any Bundle. For example it contains Bundle id[About] which is usually is used by system but as a programmer/user you are not limited on changing/reading[More]. The same as you can add K/V for your own purposes and read it in runtime. You could noticed that some frameworks forces you to add K/V into your's application to identify you or some other cases.
.entitlements is a property list with enabled capabilities(e.g. ApplePay)
[Info.plist location]
[Vocabulary]