In one of my use cases I need to provide the similar experience people have while they search their mailbox like Gmail. It can search through both email subject and body.
I have some support tickets saved in Postgres. These tickets contain user messages which we can't store in plain text. We have to encrypt the data. Now if we want to build an index for providing full text search how do we go about it considering the index can't contain the actual data
Any pointer on how Gmail or any other similar providers solve this problem would also be great.
There is no way to do that. Either data are encrypted or not. If they are encrypted, the database does not know the value, so it cannot perform full text search. That's pretty obvious.
Security comes at a price, and in this case, it is performance.
PostGreSQL is not able to encrypt data at the storage level as Oracle or Microsoft SQL Server do with Transparent Data Encryption (TDE). When Using TDE the data are encrypted in the storage files, not in memory so, you can use a fulltext search as usual.
Related
I am creating accounting/invoicing software and my database is in postgreSQL. Should I create a separate database for each user since the data is sensitive financial data? Or is having a user foreign key secure enough? If I am hosting the database on aws I understand that I could have a few db servers across multiple availability zones and regions so that if one is compromised it wouldn't effect everyone even if many users have info stored in a single database. Is this safe enough? Thanks!
In general no. Encrypt the data so that if someone exfiltrates a dump they can't actually use it without the decryption key. If you're worried that someone with admin access can see the user's information then you might want to consider a user-level encryption for all fields related to personally identifiable information.
There are few ways you could go about it but I wouldn’t create a new DB for every customers. It will be too expensive and a pain to maintain and evolve.
To me, this sounds like you are creating a multi-tenant application.
I’d personally use the row-level security feature in Postgres (see this article) or create a separate Schema for each Customer.
You can add an extra layer of protection with encryption at rest. AWS support it (link)
Good Day!
I have set up an automated daily email coming from Google Analytics and need to extract the CSV file contents and dump them into an Oracle database table on a daily basis...
I'm almost positive that this is possible, but not sure how to go about setting this up?
If someone has already set this up, could you share the steps?
thx in advance!
There's a PL/SQL Mail Client API available on source forge. It appears to be a PL/SQL wrapper for the Java Mail API, though I've never used either one.
So I'm in the middle of planning a little web app that will require quite large amounts of data stored on a user level, in one case, the system would take a large object from a system level and make a "user specific" version, a user can have multiple ones of these. Simplest would be to compare it to a form stored in a google spreadsheet, where the user is expected to use the template spreadsheet, then change not only the answers but also the question.
Security wise I am quite OK
In the second case there is requirement to store multiple objects, size about 250k to maybe 3mb, once again on a user specific level, with a potential to move it to a system level so additional users can access it. As an example, say the user can upload pictures, but may not want to share all of them. However, a user may choose to "publish" a small number of them because they are happy with those specific pictures.
What design patterns should I consider using specifically around web apps where the user have decent amounts of data? For example, would it make most sense to use a single large database and have a table that keeps track of resources or create separate tables per user?
I have considered putting it all in a mongo database.
Your approach may be wrong.
If you want to store user based binary data and make it accessible for the user itself or the community, you would need a hierarchic structure like so:
userid1
pic1,pic2,pic3
userid2
pic4,pic5,pic6
community
pic7,pic8
You could then grant read permissions to "community" for all users, and permission for each user to its own directory.
Usually there is nothing wrong using a database to store binary files if you consider partitioning, role permissions and an applicable interface to access the data.
My suggestion is to use a binary repository like Artifactory.
It provides hierarchic structures, simple search queries using HTTP requests and has caching abilities for frequently queried objects.
I also think that http requests are a lot easier to use and also there is an abstraction layer to the data which is more secure.
Artifactory is free.
There is a possibility that a user of my app may enter tens/hundreds/thousands of pieces of information into the app. For example, they may enter names, addresses, phone numbers of various people.
What would you recommend is the best way to store (a potentially large amount) these strings?
Ideally, I'd like to be able to export the inputted data as a text delimited file or a spreadsheet.
You should consider either CoreData or sqlite. CoreData can be over an xml or sqlite database. If you use sqlite, you should consider the fmdb wrapper on github
But, a lot depends on the access patterns. Sqlite is very fast but it also offers querying capabilities.
Sqlite3 is a great way to store a large amount of data.
http://www.icodeblog.com/2008/08/19/iphone-programming-tutorial-creating-a-todo-list-using-sqlite-part-1/
Failing that you could always use NSUserDefaults
We have a number of customers that we have to send monthly invoices too. Right now, I'm managing a codebase that does SQL queries against our customer database and billing database and places that data into emails - and sends it.
I grow weary of maintaining this every time we want to include a new promotion or change our customer service phone numbers. So, I'm looking for a replacement to move more of this into the hands of those requesting the changes.
In my ideal world, I need :
A WYSIWYG (man, does anyone even say that anymore?) email editor that generates templates based upon the output from a Database Query.
The ability to drag and drop various fields from the database query into the email template.
Display of sample email results with the database query.
Web application, preferably not requiring IIS.
Involve as little code as possible for the end-user, but allow basic functionality (i.e. arrays/for loops)
Either comes with it's own email delivery engine, or writes output in a way that I can easily write a Python script to deliver the email.
Support for generic Database Connectors. (I need MSSQL and MySQL)
F/OSS
So ... can anyone suggest a project like this, or some tools that'd be useful for rolling my own?
(My current alternative idea is using something like ERB or Tenjin, having them write the code, but not having live-preview for the editor would suck...)
I think your looking for a reporting tool which is also capable of sending email. Sending a generared report in html or pdf shouldn't be to hard to do as well.
I've used JasperReports in the past for which I think it should fit your needs.
Another good solution is the pentaho reporting tool
You could easily write something on your own.. give them a basic edit control and allow them to use psuedo variables like {customername} {anothercustomerattribute} within the mail body.
On submit either send directly or save as template.
When the template is sent away the script automatically parses stuff like {customername} into the real customers name from the database.
Your own very very simple custom scriptlanguage :)
Everything else like loops and so on would be maintained on serverside. And if you want particular groups of customers to receive the letter, allow the enduser to select from selectboxes or whatever and do the rest on the serverside with pre-defined rules.