I have a list of objects that will be saved locally as a .json file. The problem is that I don't want to update the .json file on each user modification of one of the lists' objects. Any window focus change will be sufficient to trigger the save() function to ensure data will not be lost.
How to implement this scenario or any other suggestions?
Related
I have 4000 files each averaging 30Kb in size landing in a folder on our on premise file system each day. I want to apply conditional logic (several and/or conditions) against details in their file names to only move files matching the conditions into another folder. I have tried linking a meta data activity which gets all files in the source folder with a filter activity which applies the conditional logic with a for each activity with an embedded copy activity. This works but it is taking hours to process the files. When running the pipeline in debug the output window appears to list each file copied as a line item. I’ve increased the batch count setting in the for each to 50 but it hasn’t improved things. Is there a way to link the filter activity directly to the copy activity without using for each activity? Ie pass the collection from the filter straight into copy’s source. Alternatively, some of our other pipelines just use the copy activity pointing to a source folder and we configure its filefilter setting with a simple regex using a combination of * and ?, which is extremely fast. However, in this particular scenario, my conditional logic is more complex and I need to compare attributes in each file’s name with values to decide if the file should be moved. The filefilter setting allows dynamic content so I could remove the filter activity completely, point the copy to the source folder and put the conditional logic in the filefilter’s dynamic content area but how would I get a reference to the file name to do the conditional checks?
Here is one solution:
Write array output as text to a .json in Blob Storage (or wherever). Here are the steps to make that work:
Copy Data Source:
Copy Data Sink:
Write the json (array output) to a text file that has the name of the files you want to copy.
Copy Activity Source (to get it from JSON to .txt):
Sink will be .txt file in your Blob.
Use that text file in your main copy activity and use the following setting:
This should copy over all the files that you identified in your Filter Activity.
I realize this is a work around, but really is the only solution for what you are asking. Otherwise there is no way to link a filter activity straight to a copy activity.
I have a reorderable list where i read the further information of each row on tap from a json file. Each row displays the corresponding json files name. I read these from a local folder in users device. The list shown here enables the user to reorder the items. The problem is I want to ensure persistent reodering by which I mean my app should remember the ordering user made next time the app is launched. I cannot think of anyway to go with this. Do I store a local json file keeping all the file names and the corresponding row index? What would be a best practice for this. This list is though to have row amount of 50 to 200 so I need a scalable solution.
I have a azure blob container where some json files with data gets put every 6 hours and I want to use Azure Data Factory to copy it to an Azure SQL DB. The file pattern for the files are like this: "customer_year_month_day_hour_min_sec.json.data.json"
The blob container also has other json data files as well so I have filter for the files in the dataset.
First question is how can I set the file path on the blob dataset to only look for the json files that I want? I tried with the wildcard *.data.json but that doesn't work. The only filename wildcard I have gotten to work is *.json
Second question is how can I copy data only from the new files (with the specific file pattern) that lands in the blob storage to Azure SQL? I have no control of the process that puts the data in the blob container and cannot move the files to another location which makes it harder.
Please help.
You could use ADF event trigger to achieve this.
Define your event trigger as 'blob created' and specify the blobPathBeginsWith and blobPathEndsWith property based on your filename pattern.
For the first question, when an event trigger fires for a specific blob, the event captures the folder path and file name of the blob into the properties #triggerBody().folderPath and #triggerBody().fileName. You need to map the properties to pipeline parameters and pass #pipeline.parameters.parameterName expression to your fileName in copy activity.
This also answers the second question, each time the trigger is fired, you'll get the fileName of the newest created files in #triggerBody().folderPath and #triggerBody().fileName.
Thanks.
I understand your situation. Seems they've used a new platform to recreate a decades old problem. :)
The patter I would setup first looks something like:
Create a Storage Account Trigger that will fire on every new file in the source container.
In the triggered Pipeline, examine the blog name to see if it fits your parameters. If no, just end, taking no action. If so, binary copy the blob to a account/container your app owns, leaving the original in place.
Create another Trigger on your container that runs the import Pipeline.
Run your import process.
Couple caveats your management has to understand. You can be very, very reliable, but cannot guarantee compliance because there is no transaction/contract between you and the source container. Also, there may be a sequence gap since a small file can usually process while a larger file is processing.
If for any reason you do miss a file, all you need to do is copy it to your container where your process will pick it up. You can load all previous blobs in the same way.
I have a cloud function which is set to be triggered by a cloud storage bucket.
Is there anyway to tell if an event is set off by a newly uploaded object or rather an overwritten object?
I tried to console.log out the event object but nothing seems to be indicative of whether or not the object has been overwritten.
I do notice that there exists an "overwroteGeneration" attribute in Cloud Pub/Sub Notifications which the trigger event here is based on, but it's not available here.
As staged by #Doug Stevenson, it looks like there is no easy way to achieve this. Moreover, I have been playing around with metageneration and, according to this example in the documentation (under "You upload a new version of the image
"), when an object is overwritten (even when versioning is enabled), it will get its own new generation and metageneration numbers, reason why, as you commented in the comment to the other answer, metageneration = 1 in such a scenario.
The only workaround I see, and which may not satisfy your specific requirements, is using two Cloud Functions (let's call them funcA and funcB), one (funcA) that identifies object creation with google.storage.object.finalize and another one (funcB) that detects object overwriting with google.storage.object.archive or google.storage.object.delete (depending if you are using versioning or not, respectively). In this case, funcA would be triggered twice, because for object overwritting it will be triggered too, but depending on your use case, you can identify the proximity in time of the create and delete events and detect that these detect to a single event:
funcA logs:
funcB logs:
I know this does not exactly solve the question you posted, but I do not think there is any way to identify, using a single Cloud Function, that an object has been either created or overwritten, it looks like you will need one Cloud Function for each of those procedures.
In order to determine the nature of the file upload (new file upload rather than existing file upload), you need to use the event and delivered to the function to figure it out.
You'll need to use an Object Finalize event type. As you can see from the documentation there:
This event is sent when a new object is created (or an existing object
is overwritten, and a new generation of that object is created) in the
bucket.
What's not so clear from that doc is that the metageneration combined with the resourceState property of the event is the indicator.
The documentation here should be clear about how to use metageneration along with resourceState to determine if a change to a bucket is a new file or a replaced file:
The resourceState attribute should be paired with the 'metageneration'
attribute if you want to know if an object was just created. The
metageneration attribute is incremented whenever there's a change to
the object's metadata. For new objects, the metageneration value is 1.
I have a CheckBox in my TabPage on my Form, if I select the checkBox, the value is saved in a Table field (present in my FormDataSource: ParametersTable).
I want to refresh the form when I enter in TabPage, just Like pressing F5.
Is it possible?
There is a great article about different methods of refreshing the form's data here. Here are the basic outline:
1. Refresh
This method basically refreshes the data displayed in the form controls with whatever is stored in the form cache for that particular datasource record. Calling refresh() method will NOT reread the record from the database. So if changes happened to the record in another process, these will not be shown after executing refresh().
2. Reread
Calling reread() will query the database and re-read the current record contents into the datasource form cache. This will not display the changes on the form until a redraw of the grid contents happens (for example, when you navigate away from the row or re-open the form).
You should not use it to refresh the form data if you have through code added or removed records.
3. Research
Calling research() will rerun the existing form query against the database, therefore updating the list with new/removed records as well as updating all existing rows. This will honor any existing filters and sorting on the form, that were set by the user.
4. ExecuteQuery
Calling executeQuery() will also rerun the query and update/add/delete the rows in the grid. ExecuteQuery should be used if you have modified the query in your code and need to refresh the form to display the data based on the updated query.
I strongly recommand that you read the article. Try to use some of the methods above or some combinations of them.
Start with research() method, it might solve your problem:
formDataSource.research();