I have a set of tweets data in MongoDB. all tweets contain some bad expression like //,? or links. for example:
"text": "We\\'re through one end in 🇦🇺 Australia vs. 🏴 Scotland in the GOLD medal match of men\\'s triples! Get around the action 👉 https:12345567 #GC2018 #ShareTheDream https
How can I replace (remove) them? I tried to use remove() function but removing it is not good because it will remove the whole text. I just want to remove the bad words, but leave the text as it is
For special characters use str.replace(/[^a-zA-Z0-9]/g, "");.
Or you can specifiy each char you wish to remove string.replace(/[&\/\\#,+()$~%.'":*?<>{}]/g, '');
For URL's b = url.replace(/(?:https?|ftp):\/\/[\n\S]+/g, '');
Pay attention to escape character '\'.
Related
I've run into an interesting problem I'm hoping someone can shed some light on.
I'm trying to pull a unique list of names from an MS SQL Database - but the company has been sloppy with their names. They were tacking on a code to the end of last name for some users. I need to remove that code.
Example:
firstname lastname
John Doe
Mary Smith AST
Mike Jackson AST
Brian Astor
Jackie Masterson
In the example, "AST" is the code they tack on. It's not tacked on to all last names either. I need to get an output of just the last names without the code.
I would have expected this is a simple use of REPLACE. I tried:
select REPLACE(lastname, ' AST', '') from table
Note the leading space in the quotes for the search phrase... this does work to remove the "AST" appended to the last names.
However - my problem is that it will also remove anywhere AST appears at the BEGINNING of the field. So Brian Astor comes out as "Brian or" since the field started with AST. However... it correctly does not remove ast from the middle, so Jackie Masterson is fine.
Any ideas why it is ignoring the leading space in my search phrase for the beginning of the field? I've tried ltrim to eliminate the possibility the field has leading spaces.
Thanks!
Replace with an empty string will eliminate the searched string anywhere in your source string. So the behaviour is as expected.
If you only need to replace ' ast' at the end of your searched string, try something like this:
select replace(lastname + '$$$', ' AST$$$', '') from table
Of course you need to be sure that the $$$ appended don't appear by chance in your source string (lastname). Which I guess is not that likely.
I'm coming across a strange situation where I cannot search on string tags that end with a special character. So far I've tried ) and ].
For example, given a Fruit index with a record with a tag apple (red), if you query (using the JS library) with tagFilters: "apple (red)", no results will be returned even if there are records with this tag.
However, if you change the tag to apple (red (not ending with a special character), results will be returned.
Is this a known issue? Is there a way to get around this?
EDIT
I saw this FAQ on special characters. However, it seems as though even if I set () as separator characters to index that only effects the direct attriubtes that are searchable, not the tag. is this correct? can I change the separator characters to index on tags?
You should try using the array syntax for your tags:
tagFilters: ["apple (red)"]
The reason it is currently failing is because of the syntax of tagFilters. When you pass a string, it tries to parse it using a special syntax, documented here, where commas mean "AND" and parentheses delimit an "OR" group.
By the way, tagFilters is now deprecated for a much clearer syntax available with the filters parameter. For your specific example, you'd use it this way:
filters: '_tags:"apple (red)"'
I'm working to build an import tool that utilizes a quoted CSV file. However, several of the fields in the CSV file are reported as such:
"=""38000"""
Where 38000 is the data I need. The data integration software I use (Talend 6.11) already strips the leading and trailing double quotes for me (so, "38000" becomes 38000), but I can't find a way to get rid of those others.
So, essentially, I need "=""38000""" to become "38000" where the leading "=" is removed and the trailing "" is removed.
Is there a TRIM function that can accomplish this for me? Perhaps there is a method in Talend that can do this?
As the other answer stated, you could do that operation in SQL. Or, you could do it in Java, Groovy, etc, within Talend. However, if there is an existing Talend component which does the job, my preference is to use it. That leads to faster development, potentially less testing, and easier maintenance. Having said that, it is important to review all the components which are available, so you know what's available to you.
You can use the Talend component tReplace, to inspect each of the input columns you want to trim of quotes and equal signs. A single tReplace component can do search and replace operations on multiple input columns. If all the of the replaces are related to each other, I would keep them within a single tReplace. When it gets to the point of doing unrelated replacements, I might place those within a new tReplace so that logical operations are organized and grouped together.
tReplace
For a given Input Column
search for "=", replace with ""
search for "\"", replace with ""
Something like that:
SELECT format( '"%s"', trim( both '"=' from '"=""38000"""' ) );
-[ RECORD 1 ]---
format | "38000"
1st: trim() function removes all " and = chars. Result is simply 38000
2nd: with format can add double quote back to get wishful end result
Alternatively, can use regexp and other Postgres string functions.
See more:
https://www.postgresql.org/docs/current/static/functions-string.html
I call my statement with CONTAINS function, but sometimes it does not return correct records, e.g. I want to return row which contain in one field word 'Your':
SELECT [Email]
,[Comment]
FROM [USERS]
WHERE CONTAINS(Comment, 'Your')
It gives mi 0 result despite that this field contains this word (the same with 'as', 'to', 'was', 'me'). When I use 'given' instead of 'Your' then I receive a result. Is there maybe a list of words which cannot be used with CONTAINS? Or maybe this words are to short (when i use 'name' then i receive the results)? The work 'Your' is at the beginning in field Comment.
The field is of type 'text' and has enabled full-text index.
Words such as those you mention are "stop words"; they are expressly excluded from being indexed and searched in Full Text Search due to how common (and thereby meaningless for searches) they are. You'll notice the same thing when searching Google, for instance.
It is possible to edit the list, but I would avoid doing so except perhaps to add words to it; the words in the list are chosen very well, IMHO, for their lack of utility in searches.
I have an app that utilizes hashtags to help tag posts. I am trying to have a more detailed search.
Lets say one of the records I'm searching is:
The #bird flew very far.
When I search for "flew", "fle", or "#bird", it should return the record.
However, when I search "#bir", it should NOT return the sentence because the whole the tag being searched for doesn't match.
I'm also not sure if "bird" should even return the sentence. I'd be interested how to do that though as well.
Right now, I have a very basic search:
SELECT "posts".* FROM "posts" WHERE (body LIKE '%search%')
Any ideas?
You could do this with LIKE but it would be rather hideous, regexes will serve you better here. If you want to ignore the hashes then a simple search like this will do the trick:
WHERE body ~ E'\\mbird\M''
That would find 'The bird flew very far.' and 'The #bird flew very far.'. You'd want to strip off any #s before search though as this:
WHERE body ~ E'\\m#bird\M''
wouldn't find either of those results due to the nature of \m and \M.
If you don't want to ignore #s in body then you'd have to expand and modify the \m and \M shortcuts yourself with something like this:
WHERE body ~ E'(^|[^\\w#])#bird($|[^\\w#])'
-- search term goes here^^^^^
Using E'(^|[^\\w#])#bird($|[^\\w#])' would find 'The #bird flew very far.' but not 'The bird flew very far.' whereas E'(^|[^\\w#])bird($|[^\\w#])' would find 'The bird flew very far.' but not 'The #bird flew very far.'. You might also want to look at \A instead of ^ and \Z instead of $ as there are subtle differences but I think $ and ^ would be what you want.
You should keep in mind that none of these regex searches (or your LIKE search for that matter) will uses indexes so you're setting yourself up for lots of table scans and performance problems unless you can restrict the searches using something that will use an index. You might want to look at a full-text search solution instead.
It might help to parse the hash tags out of the text and store them in an array in a separate column called say hashtags when the articles are inserted/updated. Remove them from the article body before feeding it into to_tsvector and store the tsvector in a column of the table. Then use:
WHERE body_tsvector ## to_tsquery('search') OR 'search' IN hashtags
You could use a trigger on the table to maintain the hashtags column and the body_tsvector stripped of hash tags, so that the application doesn't have to do the work. Parse them out of the text when entries are INSERTed or UPDATEd.