I want to get all audit entries from an Alfresco audit application.
Following the Alfresco Audit REST API documentation, I wrote this query:
curl -u admin:admin "http://localhost:8080/alfresco/service/api/audit/query/audit-custom?fromId=1&toId=100000&fromTime=2017-02-19T18:46:09.212+09:00&toTime=2017-02-20T18:46:09.212+09:00&user=admin&forward=true&limit=0&verbose=true"
(Note the fromTime={2017-02-19T18:46:09.212+09:00}&toTime={2017-02-20T18:46:09.212+09:00} part)
PROBLEM: This returns the exact same 90 results as the request without the fromTime and toTime parameters. By the way, I installed Alfresco on 2017-02-21 so the query above should actually return zero results.
Same problem with { and } characters around the dates
Same problem with simpler dates like fromTime={2017-02-19}&toTime={2017-02-20}
Same problem without the extra parameters
How to use correctly the fromTime and toTime parameters?
It is not obvious in the documentation, but fromTime and longTime must be entered as long values.
Here is the relevant Alfresco source code:
public static final String PARAM_FROM_TIME = "fromTime";
protected Long getParamFromTime(WebScriptRequest req)
{
String paramStr = req.getParameter(PARAM_FROM_TIME);
try
{
return Long.parseLong(paramStr);
}
catch (NumberFormatException e)
{
return DEFAULT_TO_TIME;
}
}
As you can see, format problems are quietly dropped, and default values used instead, which is what is happening with the requests included in the question above.
With long values, a valid request could be:
curl -u admin:admin "http://localhost:8080/alfresco/service/api/audit/query/audit-custom?fromTime=1485152993728&toTime=1485239393728&verbose=true"
(Note the fromTime=1485152993728&toTime=1485239393728 part)
Here is a convenient Java snippet to convert dates to long values when performing tests:
import java.util.concurrent.TimeUnit;
public class Main {
public static void main(String[] args) {
long now = System.currentTimeMillis();
long lastMonth = now - TimeUnit.DAYS.toMillis(30);
long lastMonthPlusOne = now - TimeUnit.DAYS.toMillis(29);
System.out.println("Now:" + now);
System.out.println("Last month:" + lastMonth);
System.out.println("Last month plus one day:" + lastMonthPlusOne);
}
}
Related
I have a folder structure of the type year/month/day/hour/*, and I'd like the beam to read this as an unbounded source in chronological order. Specifically, this means reading in all the files in the first hour on record and adding their contents for processing. Then, add the file contents of the next hour for processing, up until the current time where it waits for new files to arrive in the latest year/month/day/hour folder.
Is it possible to do this with apache beam?
So what I would do is to add timestamps to each element according to the file path. As a test I used the following example.
First of all, as explained in this answer, you can use FileIO to match continuously a file pattern. This will help as, per your use case, once you have finished with the backfill you want to keep reading new arriving files within the same job. In this case I provide gs://BUCKET_NAME/data/** because my files will be like gs://BUCKET_NAME/data/year/month/day/hour/filename.extension:
p
.apply(FileIO.match()
.filepattern(inputPath)
.continuously(
// Check for new files every minute
Duration.standardMinutes(1),
// Never stop checking for new files
Watch.Growth.<String>never()))
.apply(FileIO.readMatches())
Watch frequency and timeout can be adjusted at will.
Then, in the next step we'll receive the matched file. I will use ReadableFile.getMetadata().resourceId() to get the full path and split it by "/" to build the corresponding timestamp. I round it to the hour and do not account for timezone correction here. With readFullyAsUTF8String we'll read the whole file (be careful if the whole file does not fit into memory, it is recommended to shard your input if needed) and split it into lines. With ProcessContext.outputWithTimestamp we'll emit downstream a KV of filename and line (the filename is not needed anymore but it will help to see where each file comes from) and the timestamp derived from the path. Note that we're shifting timestamps "back in time" so this can mess up with the watermark heuristics and you will get a message such as:
Cannot output with timestamp 2019-03-17T00:00:00.000Z. Output timestamps must be no earlier than the timestamp of the current input (2019-06-05T15:41:29.645Z) minus the allowed skew (0 milliseconds). See the DoFn#getAllowedTimestampSkew() Javadoc for details on changing the allowed skew.
To overcome this I set getAllowedTimestampSkew to Long.MAX_VALUE but take into account that this is deprecated. ParDo code:
.apply("Add Timestamps", ParDo.of(new DoFn<ReadableFile, KV<String, String>>() {
#Override
public Duration getAllowedTimestampSkew() {
return new Duration(Long.MAX_VALUE);
}
#ProcessElement
public void processElement(ProcessContext c) {
ReadableFile file = c.element();
String fileName = file.getMetadata().resourceId().toString();
String lines[];
String[] dateFields = fileName.split("/");
Integer numElements = dateFields.length;
String hour = dateFields[numElements - 2];
String day = dateFields[numElements - 3];
String month = dateFields[numElements - 4];
String year = dateFields[numElements - 5];
String ts = String.format("%s-%s-%s %s:00:00", year, month, day, hour);
Log.info(ts);
try{
lines = file.readFullyAsUTF8String().split("\n");
for (String line : lines) {
c.outputWithTimestamp(KV.of(fileName, line), new Instant(dateTimeFormat.parseMillis(ts)));
}
}
catch(IOException e){
Log.info("failed");
}
}}))
Finally, I window into 1-hour FixedWindows and log the results:
.apply(Window
.<KV<String,String>>into(FixedWindows.of(Duration.standardHours(1)))
.triggering(AfterWatermark.pastEndOfWindow())
.discardingFiredPanes()
.withAllowedLateness(Duration.ZERO))
.apply("Log results", ParDo.of(new DoFn<KV<String, String>, Void>() {
#ProcessElement
public void processElement(ProcessContext c, BoundedWindow window) {
String file = c.element().getKey();
String value = c.element().getValue();
String eventTime = c.timestamp().toString();
String logString = String.format("File=%s, Line=%s, Event Time=%s, Window=%s", file, value, eventTime, window.toString());
Log.info(logString);
}
}));
For me it worked with .withAllowedLateness(Duration.ZERO) but depending on the order you might need to set it. Keep in mind that a value too high will cause windows to be open for longer and use more persistent storage.
I set the $BUCKET and $PROJECT variables and I just upload two files:
gsutil cp file1 gs://$BUCKET/data/2019/03/17/00/
gsutil cp file2 gs://$BUCKET/data/2019/03/18/22/
And run the job with:
mvn -Pdataflow-runner compile -e exec:java \
-Dexec.mainClass=com.dataflow.samples.ChronologicalOrder \
-Dexec.args="--project=$PROJECT \
--path=gs://$BUCKET/data/** \
--stagingLocation=gs://$BUCKET/staging/ \
--runner=DataflowRunner"
Results:
Full code
Let me know how this works. This was just an example to get started and you might need to adjust windowing and triggering strategies, lateness, etc to suit your use case
I am trying to save this data into oracle datebase but (Member start Date and Member end date) giving an error i am using datepicker from JPallet for date and preparedstatemet allows me in pst.setString(6, startdate.getDate()); int or string. Please help me for solving this problem. I shall be very thankful to you.
Error Lines:
pst.setString(6, startdate.getDate());
pst.setString(7, enddate.getDate());
Full method:
private void jButton1ActionPerformed(java.awt.event.ActionEvent evt) {
con = javadb.ConnectDb();
try{
String sql = "insert into membertable(id,name,membershipno,department,idno,"
+ "startdate,enddate,officeaddress,officephone,homeaddress,homephone,dateofbirth,"
+ "fieldofinterest,remarks) values(?,?,?,?,?,?,?,?,?,?,?,?,?,?)";
pst = (OraclePreparedStatement) con.prepareStatement(sql);
pst.setString(1, null);
pst.setString(2, name.getText());
pst.setString(3, membershipno.getText());
pst.setString(4, department.getText());
pst.setString(5, idno.getText());
pst.setString(6, startdate.getDate());
pst.setString(7, enddate.getDate());
pst.setString(8, officeaddress.getText());
pst.setString(9, officephone.getText());
pst.setString(10, homeaddress.getText());
pst.setString(11, homephone.getText());
pst.setString(12, dateofbirth.getText());
pst.setString(13, fieldofinterest.getText());
pst.setString(14, remarks.getText());
pst.executeUpdate();
JOptionPane.showMessageDialog(null, "Book added successfully!");
}catch(SQLException e){
JOptionPane.showMessageDialog(null, e);
}
update_table();
}
And date picker code is here
private void startdateActionPerformed(java.awt.event.ActionEvent evt) {
}
And this is not the textfield so therefore i am facing the problem. And i want to correct and also don't want to change it because it is front end reqiurement. Because user don't wants to insert date manually. Thanks.
When inserting a date, don't use pst.setString(). Instead, you should use pst.setDate() (this is really the whole point of using PreparedStatements in the first place). Because setString is inserting a date string with some unknown format. setDate tells the database to expect a date and there is no date format conversion necessary.
Also, to do this, you'll need to use a java.sql.Date instead of (what I'm assuming your palette is returning) a java.util.Date.
So this
pst.setString(6, startdate.getDate());
pst.setString(7, enddate.getDate());
should look like this
pst.setDate(6, new java.sql.Date(startdate.getDate().getTime()));
pst.setDate(7, new java.sql.Date(enddate.getDate().getTime()));
I've noticed that if a date property comes back from the server with the value "2013-07-11T17:11:04.700", then breeze changes the value to Thu Jul 11 19:11:04 UTC+0200 2013.
Notice the time is now 2 hours ahead !
I had already come across this issue when saving entities, so I had to explicitly convert my date properties using momentjs :
date.hours(date.hours() - moment().zone() / 60);
But now it seems the problem occurs also when doing read operations.
What's the best way to make sure breeze does not alter values of my date properties ?
Breeze does not manipulate the datetimes going to and from the server in any way EXCEPT to add a UTZ timezone specifier to any dates returned from the server that do not already have one. This is only done because different browsers interpret dates without a timezone specifier differently and we want consistency between browsers.
The source of your issues is likely to be that when you save your data with dates to the database, that the dateTime datatype you are using does NOT contain a timezone offset. This means that when the data is retrieved you are likely "losing" the offset and the Breeze default mentioned above kicks in. This can be corrected by using a database date time datatype with an timezone offset ( datetime2 or datetimeoffset in SQLServer).
Note that your browser DOES format dates according to it's current timezone.
Another approach is that you can replace Breeze's DataType.parseDateFromServer to NOT infer any time zone info if it is not provided:
breeze.DataType.parseDateFromServer = function (source) {
return new Date(Date.parse(source));
};
However, this can run into the problem that different browsers interpret DateTime strings without a time zone offset differently... So you may still get strange results depending on the browser. If that happens you will need to add some browser detection code to the snippet above.
Another alternative is to do the following using the moment.js library.
breeze.DataType.parseDateFromServer = function (source) {
var date = moment(source);
return date.toDate();
};
Not sure how helpful this is, but hopefully Breeze's behavior is clearer.
By default, Breeze does not provide any way to do this, but you can keep the below code in your model JS file to overcome this issue:
breeze.DataType.parseDateFromServer = function (source) {
if (typeof source === 'string') {
//Check for local offset time or UTC time from server
if (source.slice(-1) !== "Z") {
var oldSource = source;
try {
source = source.substring(0, source.lastIndexOf("-") - 1)
source = new Date(source);
var tzDifference = source.getTimezoneOffset();
//convert the offset to milliseconds, add to targetTime, and make a new Date
var offsetTime = new Date(source.getTime() + tzDifference * 60 * 1000);
return offsetTime;
}
catch (err) {
source = new Date(source);
return source;
}
}
else {
source = new Date(source);
var tzDifference = source.getTimezoneOffset();
//convert the offset to milliseconds, add to targetTime, and make a new Date
var offsetTime = new Date(source.getTime() + tzDifference * 60 * 1000);
return offsetTime;
}
}
}
I want to know whether is it possible to reject numeric phrases or numeric values while indexing or searching in Lucene.net.
For example (this is one line),
Hi all my no is 4756396
Now, when I index or search it should reject the numeric value 4756396 to be indexed or searched. I tried making a custom stop word list with 1, 2, 3, 4, 5, 6, etc, but I guess it will only ignore if a single number will appears.
You can copy the StandardAnalyzer and customize the grammar (simple JFlex stuff) to reject number tokens. If you do that, you'll need to port back the analyzer to Java since JFlex will generate java code, tho you could give it a try with C# Flex.
You could also write a TokenFilter that scans tokens one by one and rejects them if they are numbers. If you wanna filter only whole numbers and still retain numbers that are for example separate by hyphens, the filter could simply attempt a double.TryParse() and if it fails you accept the Token. A more robust and customizable solution would still use a lexical parser.
Edit:
Heres a quick sample of what I mean, with a little main method that shows how to use it. In this I used a TryParse() to filter out tokens, if it were for a more complex production system I'd use a lexical parser system. (take a look at C# Flex for that)
public class NumericFilter : TokenFilter
{
private ITermAttribute termAtt ;
public NumericFilter(TokenStream tokStream)
: base(tokStream)
{
termAtt = AddAttribute<ITermAttribute>();
}
public override bool IncrementToken()
{
while (base.input.IncrementToken())
{
string term = termAtt.Term;
double res ;
if(double.TryParse(term, out res))
{
// skip this token
continue;
}
// accept this token
return true;
}
// no more token in the stream
return false;
}
}
static void Main(string[] args)
{
RAMDirectory dir = new RAMDirectory();
IndexWriter iw = new IndexWriter(dir, new KeywordAnalyzer(), IndexWriter.MaxFieldLength.UNLIMITED);
Document d = new Document();
Field f = new Field("text", "", Field.Store.YES, Field.Index.ANALYZED);
d.Add(f);
// use our Filter here
f.SetTokenStream(new NumericFilter(new LowerCaseFilter(new WhitespaceTokenizer(new StringReader("I have 300 dollars")))));
iw.AddDocument(d);
iw.Commit();
IndexReader reader = iw.GetReader();
// print all terms in the text field
TermEnum terms = reader.Terms(new Term("text", ""));
do
{
Console.WriteLine(terms.Term.Text);
}
while (terms.Next());
reader.Dispose();
iw.Dispose();
Console.ReadLine();
Environment.Exit(42);
}
I'm using GWT on the client side. I store String objects in an ArrayList instance, and when I add the value "AT&T", it only seems to be storing "AT". I suspect this is the result of the ArrayList truncating the text due to the ampersand, but I have yet to write a smaller proof-of-concept to be sure. Has anyone else encountered this problem?
GWT 2.1.1
Tomcat 7.0.5
Firefox 5.0.1
So this test succeeds:
ArrayList<String> test = new ArrayList<String>();
test.add( "testing&123");
Window.alert( test.get(0) + " - " + test.get(0).contains("&") );
My code effectively does this:
String test = "AT&T";
MyApp.getInstance().getDataStore().add( test );
Window.alert( test + " - " + MyApp.getInstance().getDataStore().getItems().get(0) );
public void DataStore.add( String item ) {
itemsList.add( item );
}
public ArrayList<String> getItems() {
return itemList;
}
The output is "AT&T - AT". So if the ArrayList is not the problem, is it the method calls?
Well, it turns out it was an interaction between my History Manager and the DataStore class. A very strange situation that no one could have helped me with given the limited amount of information I had provided.