I need to implement an in-application message-box system (like any forum/social network has).
Number of users is in the tens of thousands and up.
I am thinking to do it using an actual email server and write a client for it inside the application (or just pick an open source client and adjust it to appear like a private messages system).
Or
Should I just do it as an integral part of my application (DB, server side and all)?
Which approach is the correct one?
(I do not think this is a consideration, but the bulk of the application is written in PHP).
Use your application back-end.
Email (SMTP/IMAP) sounds simple but has way too much administration and programming overhead if you do it properly - or you get random problems.
If you need to send emails to your users (which you shouldn't need to - I quite like how Stackoverflow works for example in that respect) then just be a client to email (SMTP).
Another issue with 'email' is it's an old protocol which does not fit into todays needs. Hence we have new protocols and applications, like XMPP and Google Wave.
Related
I have a flask application that I want to extend with a function that allows two separate, registered users (let's call them Alice and Bob) to communicate via e-mail.
The detour via server is necessary because it shouldn't be known to Bob that Alice is the one sending the mail (so the server acts as intermediary between the two). However ideally the server shouldn't be able to read the message contents of the mail either.
Is there an easy out-of-the-box solution to this requirement? I figure that the encryption has to happen on the clients (i.e. e2ee), so flask doesn't really factor into it. Do JINJA2 or JavaScript have functions to facilitate that? Is there a best practise for this scenario or am I overthinking it?
I apologise for the lack of specific example and I hope this kind of question is alright for stackoverflow.
I'm a PHP developer with a lack of experience on other sever side langages.
I’d like you to give me leads, advice, keywords or whatever that could help me refine my research better.
What I want to do is basically to create a one to one mobile app chat that will scale.
There will be channels of 3 users: User A, User B and the "computer" pushing some messages according to some channels informations like the last time a message has been sent, etc.
User A should know if User B is online, writing, etc.
Every conversation should be stored in a database and would be analyzed by some algorithms. Those algorithms would also analyzed stuffs on user Facebook open graph.
The application should be able to send notification on IOS and Android.
A web administration should allow admin to set some stuff that will define what kind of message would be sent by the "computer".
I'v read lot of posts about websocket, xmpp, node.js, socket.io, etc but I don't have enough knowledge in those areas to decide what architecture should I build to make everything works together. Or maybe there is some cloud base solutions that would fit my needs...
Thanks
As you've stated there are many ways to implement that kind of structure but I am going to write about node.js + socket.io part;
1) It is scalable. You can use cluster, nginx, haproxy. etc. to apply load balancing to your socket.io application (see here) Of course you've to use redis or mongo or some kind of store for socket.io that different servers and processes can communicate each other. (see here)
2) socket.io have rooms. That means clients and any computer bots can join that room to share events with each other. So, in your scenario User A, User B and a computer bot should join to same room and events sent to that room will be broadcasted to every room member. (events can vary as online, typing, new message, anything) (see here)
3) node.js can send push notifications both for iOS and Android.
4) You can write every message to database of your choice on new message event.
5) You can create a REST api with Express framework for your Administration page. And you can use passport for authentication and authorization purposes. You can consume the rest api with Jquery or some other frontend framework like React etc.
Meteor is very well suited for something like this and gives you everything you need. There are also open sourced chat systems built with meteor already to get an idea of where you need to go. The more custom route would be to do what #cdagli said.
I need to stream data from a web server to clients. The data is location data that is collected and stored on the server. The clients will click a button on an html page to 'opt in' to start receiving the data. This data is never ending and there is at least one of the clients that needs to receive the data 24-7, with as few breaks as possible. The data being streamed will be client specific, as each client wont receive the exact same data.
I've done several multi-threaded tcp servers over sockets, and websockets are the way I would like to attack this, but the requirements are that this has to work in ie9.
The initial requirement was that this be a vb.net cgi executable - but during testing, I havent been able to 'use' the stream from the vb.net executable until the app finishes - like it wasn't able to flush the stdout even though I was specificly using the console.out.flush(). So If this isn't a viable option, and I can support this with facts, then I can get this requirement changed.
I've also read quite a bit about using a third party server to stream the data like Orbit and APE I think was a couple of them, but requirements are for 1 server - the web server. No other hardware can be required.
I'm pretty sure the vb.net CGI isn't the ideal solution based on what i've found, but is it doable or do I need to abandon that solution and move on to a newer technology , ISAPI? Any ideas or suggestions, even if they just point me in the right direction, are greatly appreciated.
You might go few ways.
If you would go C# .Net, then you might look into Silverlight solution. But it requires plugin in browser to be installed (like Flash). Good thing here, is that you are able to send data through normal sockets, in pure realtime from server. In same time Silverlight uses .Net so it makes some code to be shared. That helps development process. As well the way it will work in different browsers will be same.
You might have a look in similar solution using Java Applet with Java backend (can be even .Net, but again, easier to develop when both in same language).
Another option is to have fron-end using WebSockets, but as you know its not supported in IE9 and below (IE10 promises to be), and Opera is not supporting it as well.
Backend can be done in what you prefer. But bear in mind that WebSockets uses framing, and for constant but little packets its not efficient, because if you send 10 bytes, then it will create frame 2-12 bytes, and TCP packet header that is 40 bytes in average.
To support older browsers you might have a look in long-polling, but it is not as reliable as websockets.
As well it is important to calculate the amount of data and approximate amount of users that will use your system. Based on calculations you will have approximate information about how real it is, and what server will be required to handle.
I'm a real Erlang newbie (started 1 week ago), and I'm trying to learn this language by creating a small but efficient chat server. (When I say efficient I mean I have 5 servers used to stress test this with hundreds of thousands connected client - A million would be great !)
I have find some tutorials doing so, the only thing is, that every tutorial i found, are IRC like. If one user send a message, all user except sender will receive it.
I would like to change that a bit, and use one-to-one discussion.
What would be the most effective client pool for searching a connected user ?
I thought about registering the process, because it seems to do everything I need, but I really don't think this is the better way to do it. (Or most pretty way to do it anyway).
Does anyone would have any suggestions doing this ?
EDIT :
Every connected client is affected to an ID.
When the user is connected, it first send a login command to give it's id.
When an user wants to send a message to another one the message looks like this
[ID-NUMBER][Message] %% ID-NUMBER IS A FIXED LENGTH
When I ask for "the most effective client pool", I'm actually looking for the fastest way to retrieve/add/delete one client on the connected client list which could potentially be large (hundred of thousands -- maybe millions)
EDIT 2 :
For answering some questions :
I'm using Raw Socket (Using telnet right now to communicate with server) - will probably move to ssl later...
It is my own protocol
Every Client is a spawned Pid
Every Client's Pid is linked to it's own monitor (mostly for debugging reason - The client if disconnected should reconnect by it's own starting auth from scratch)
I have read a couple a book before starting coding, So I do not master yet every aspect of Erlang but I'm not unaware of it, I will read more about it when needed I guess.
What I'm really looking for is the best way to store and search thoses PIDs to send message directly from process to process.
Should I write my own search Client function using lists ?
or should I use ets ?
Or even use register/2 unregister/1 and whereis/1 to maintain my client list, using it's unique id as atom, it seems to be the simplest way to do so, I really don't know if it is efficient, but I'm pretty sure this is the ugly solution ;-) ?
I'm doing something similar to your chat program using gproc as a pubsub (similar to the demo on that page). Each client registers as it's id. To find a particular client, you do a lookup on that client id. To subscribe to a client, you add a property to that process of the client id being subscribed to. To publish, you call gproc:send(ClientId,Message). This covers your use case, the more general room based chat as well, and can handle distributed masterless registry of processes.
I haven't tested to see if it scales to millions, but it uses ets to do the storage and gproc is rock solid code by Ulf Wiger. I wouldn't count on being able to write a better implementation.
I'm also kind of new to Erlang (a couple of months), so I hope this can put you in the correct path :)
First of all, since you're a "newbie", you should know about these sites:
Erlang Official Documentation:
Most common modules are in the stdlib application, so start from there.
Alternative Documentation:
There's a real time search engine, so it is really good when searching
for specific modules.
Erlang Programming Rules:
For you to enter in the mindset of erlang programming.
Learn You Some Erlang Book:
A must read for everyone starting with Erlang. It's really comprehensive
and fun to read!
Trapexit.org:
Forum and cookbooks, to search for common problems faced by programmers.
Well, thinking about a non persistent database, I would suggest the sets or gb_sets modules (documentation here).
If you want persistence, you should try dets (see documentation above), but I can't state anything about efficiency, so you should research this topic a bit further.
In the book Learn You Some Erlang there is a chapter on data structures that says that sets are better for read intensive systems, while gb_sets is more appropriate for a balanced usage.
Now, Messaging systems are what everyone wants to do when they come to Erlang because the two naturally blend. However, there are a number of things to look into before one continues. Messaging basically involves the following things: User Registration, User Authentication, Sessions Management,Logging, Message Switching/routing e.t.c. Now, to do all or most of these, one needs to have a Database, certainly IN-MEMORY, thats leads me to either Mnesia or ETS Tables. Since you are new to Erlang, i suppose you have not yet really mastered working with these. At one moment, you will need to maintain Who is communicating with who, Who is available for Chat e.t.c. Hence you might need to look up things and write things some where.Another thing is you have not told us the Client. Is it going to be a Web Client (HTTP), is it an entirely new protocol you are implementing over raw Sockets ? Which ever way, you will need to master something called: Concurrency in Erlang. If a user connects and is assigned an ID, if your design is A process Per User, then you will have to save the Pids of these Processes or register them against some criteria, yet again monitor them if they die e.t.c. Which brings me to OTP and Supervision trees. There is quite alot, however, tell us more about the Client and Server interaction, the Network Communication you need e.t.c. Or is it just a simple Erlang RPC project you are doing for your own revision ?
EDIT Use ETS Tables, or use Mnesia RAM tables. Do not think of registering these Pids or Storing them in a list, Array or set. Look at this solution which was given to this question
I am creating a PHP system that will forward emails, (ie. someone#emailforprofession.com to someone#gmail.com) without using an SMTP server which is faster and more stable? PEAR::Mail or PHP: Mail? (or another one entirely)
I would be using extra headers and attachments but mainly trying to keep it looking and acting the same. (if anyone has an example for forwarding with attachments PLEASE provide it!)
And not exactly on the side but is there a way for people to be able to send email from they're someone#emailforprofession.com email address without having a complete inbox etc?
It would probably be better for me to use a real email provider but these are supposed to be free email addresses so paying by the email address is not an option, if anyone knows of a service that: charges by the month or year, has an API that email addresses can be created from and that can set up email forwards via the API that would be most appreciated.
I currently have a partial system together but would like to know which way I should go first.
Thank you all.
The PEAR class supports multiple mailer backends, which allows you to
switch whenever needed and you'll be able to perform some benchmarks
and see which backend suits you best. The native mail function on the
other hand is much less advanced and will show a clear difference in
performance, especially for that amount of mails.
I also suggest you have a look at PEAR's mail queue class:
http://pear.php.net/package/Mail_Queue
This is much more recommended for larger projects such as yours,
you'll also be able to implement a cron like feature which will send a
certain amount of mail per period you choose.
Here