Is creating a multi entity topic better than having all entities in separate topic? - apache-kafka

I have looked into this article and still i have some confusion regarding merging separate topics in one comprehensive topic: https://www.confluent.io/blog/put-several-event-types-kafka-topic
So i have two entities, Account & Client as given below:
Account:
"Account": {
"AccountId": "JKB123456",
"ClientId": "1234567",
"Type": "Savings",
"Currency": "USD"
}
Client:
"Client": {
"ClientId": "1234567",
"Name": "John Doe",
"PhoneNo": "777777777"
}
A Client can have at least one or more accounts associated with it. The events which will work on them ar pretty basic i.e. Create and Update.
This results in four use-cases:
Create a client - A client entity with multiple Account entities are provided. If the client with all its account is created, the use-case is considered successfully executed.
Update Client - A client (only) has some of its details updated, so a client object with the said values is provided. Account is not part of this.
Create Account - An account entity is provided to be created and associated with an existing account.
Update Account - An existing account fields can be updated in this use-case.
A create client json structure will look like this:
{
"Client": {
"ClientId": "1234567",
"Name": "John Doe",
"PhoneNo": "777777777",
"Accounts": [
{
"AccountId": "JKB123456",
"ClientId": "1234567",
"Type": "Savings",
"Currency": "USD"
},
{
"AccountId": "HKB123456",
"ClientId": "1234567",
"Type": "Savings",
"Currency": "EUR"
}
]
}
}
So should i have a single Client topic schema with list of account structure as part of it? I believe then i can publish both create and update events for the client on the same topic.
But would it be possible to publish the Account Create/update events on the same topic as well? Because i think if i do proper ordering (probably on the basis of clientid) then i can also have the all events of the same client on the same partition.
Do you think this is a good approach?

Related

what is the best model to save data on elasticsearch?

I have a rails application and use elastic search as a search engine in my rails app. this app collects data from the mobile application and could collect from any kind of mobile app. mobile app sends two types of data user profile details and user actions details. my app admins could search over this data with multiple conditions and operations and fetch the specific results and which are user profile details. after that my app admins could communicate with this profile, for example, send an email, SMS, or even chat online. In my case I have two options to save user data; first of all, I want to save user profiles details and user action details in a separate document with this structure profile doc:
POST profilee-2022-06-09/_doc
{
"profile": {
"app_id": "abbccddeeff",
"profile_id": "2faae1d6-5875-4b36-b119-74a14589c841",
"whatsapp_number": "whatsapp:+61478421940",
"phone": "+61478421940",
"email": "user#mail.com",
"first_name": "john",
"last_name": "doe"
}
}
user actions details:
POST events_app_id_2022-05-17/_doc
{
"app_id": "9vlgwrr6rg",
"event": "Email_Sign_Up",
"profile_id": "2faae1d6-5875-4b36-b119-74a14589c840",
"media": "x1z1",
"date_time": "2022-05-17T11:48:02.511Z",
"device_id": "2faae1d6-5875-4b36-b119-74a14589c840",
"lib": "android",
"lib_version": "1.0.0",
"os": "Android",
"os_version": "12",
"manufacturer": "Google",
"brand": "google",
"model": "sdk_gphone64_arm64",
"google_play_services": "available",
"screen_dpi": 440,
"screen_height": 2296,
"screen_width": 1080,
"app_version_string": "1.0",
"app_build_number": 1,
"has_nfc": false,
"has_telephone": true,
"carrier": "T-Mobile",
"wifi": true,
"bluetooth_version": "ble",
"session_id": "b1ad31ab-d440-435f-ac12-3d03c30ac44f",
"insert_id": "1e285b51-abcf-46ae-8359-9a9d58970cdf"
}
As I said before app admins search over this document to fetch specific profiles and use that result to communicate with them, in this case, the problem is the mobile user could create a profile and a few days or a few months later create some actions so user profile details and user action details are generated in different days so if app admins want to fetch specific result from this data and wrote some complex query I have at least two queries by application on my elastic search in my app it's impossible because each query must save for later use by admin, so As a result of business logic it's impossible to me, and I have to add in some case I need to implement join query that based on elastic search documentation It has cost so it's impossible In the second scenario I decided to save both user profile and action in one docs somethings like this:
POST profilee-2022-06-09/_doc
{
"profile": {
"app_id": "abbccddeeff",
"profile_id": "urm-2faae1d6-5875-4b36-b119-74a14589c841",
"whatsapp_number": "whatsapp:+61478421940",
"phone": "+61478421940",
"email": "user#mail.com",
"first_name": "john",
"last_name": "doe",
"events": [
{
"app_id": "abbccddeeff",
"event": "sign_in",
"profile_id": "urm-2faae1d6-5875-4b36-b119-74a14589c841",
"media": "x1z1",
"date_time": "2022-06-06T11:52:02.511Z"
},
{
"app_id": "abbccddeeff",
"event": "course_begin",
"profile_id": "urm-2faae1d6-5875-4b36-b119-74a14589c841",
"media": "x1z1",
"date_time": "2022-06-06T11:56:02.511Z"
},
{
"app_id": "abbccddeeff",
"event": "payment",
"profile_id": "urm-2faae1d6-5875-4b36-b119-74a14589c841",
"media": "x1z1",
"date_time": "2022-06-06T11:58:02.511Z"
}
]
}
}
In this case, In the same state, I have to do as same as I do in before and I have to generate a profile index per day and append user action to it, so It means I have to update continuously each day, assume I have 100,000 profile and each one have 50 actions it means 100,000 * 50 per day update that have severity on my server so still it's impossible. So Could you please help me what is the best model to save my data in elastic search based on my descriptions?
Update: Does elastic search useful for my requirements? If I switch to other databases like MongoDB or add Hadoop it be more useful in my case?

how to divide entities and share it in clean architecture

For my new project in flutter, I am trying to follow the Clean Architecture and diving each different feature in domain, data and presentation`.
Now, I have initiated with the Authentication feature where I have started creating the Entities however, I am pretty much confused and stuck on how to modularize the code based on the Clean Architecture practice.
For example,
The response of my login service is as follow,
so do I need to create entities and models for all JSON nested objects like for response , data , user , accountData and permissions ?
or is there any way to use IResponse to store common elements like status , message and data and then use only for relevant feature.
Not sure whether it is allowed to share entities in between the features. Like user block below can be a feature in Authorization and Employee
{
"status": "success",
"message": "successfully login",
"data": {
"token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6NywiaXNMb2dnZWRJbiI6dHJ1ZSwidGVuYW50SWQiOjEyLCJpYXQiOjE2MjU5OTE5MTAsImV4cCI6MTYyNjA3ODMxMH0.I0z9OIDnQS-MI1ya6usqycoryZ1TBwj3K52BRfrpMuY",
"user": {
"user_id": 7,
"email": "jd#gmail.com",
"first_name": "John",
"last_name": "Doe",
"phone": "",
"date_of_birth": "2015-08-06T00:00:00.000Z",
},
"accountData": {
"name": "Amazon",
"account_id": 1
},
"permissions": [
"ViewAllEmployee",
"AddNewEmployee"
]
}
}

How to Create a Ticket That Belongs To a Process OTRS 5 - REST

Context
I'm developing a custom service in a .Net MVC Web Application that will connect to an OTRS web service to create/list/update tickets.
We are implementing many process workflows to make things works more efficiently.
Problem
I cannot find a way to "attach" a new ticket to a process, I know how to create a normal ticket, but not a process ticket.
I found a perl script that seems to do what I need to, but I cannot find a way to connect the problem with the solution.
Perl Script
ProcessTicketProcessSet()
Set Ticket's ProcessEntityID
my $Success = $ProcessObject->ProcessTicketProcessSet(
ProcessEntityID => 'P1',
TicketID => 123,
UserID => 123,
);
Returns:
$Success = 1; # undef
1 if setting the Activity was executed
undef if setting failed
Normal Ticket
URL:
http://someDomain.com.br/otrs/nph-genericinterface.pl/Webservice/SomeWebServiceName/Ticket?UserLogin=user&Password=abcd
Method: POST
Body:
{
"UserLogin": "user",
"Password": "abcd",
"Ticket": {
"Title": "REST - To Create Ticket",
"Type": "Unclassified",
"QueueID": "5",
"State": "new",
"Priority": "3 normal",
"CustomerUser": "someuser#someemail.com.br"
},
"DynamicField": [{
"Name": "CustomFieldOne",
"Value": "value1"
},
{
"Name": "CustomFieldTwo",
"Value": "value2"
}
],
"Article": {
"Subject": "Rest - Article Ticket",
"Body": "Test Article Creation",
"ContentType": "text/plain; charset=utf8"
}
}
How can I create a ticket that belongs to a process?
To create a ticket which belongs to a process you need to set two dynamic fields of a ticket.
ProcessManagementProcessID (which is representing the process)
ProcessManagementActivityID (which is representing the activity step of the process)
In case you also can set both dynamic fields later to set the process.
In case you do not know what values you need to set, just launch a process ticket via the UI and check via the ticket histories what values are set for both dynamic fields.

REST: update a resource with different fields requiring different user permissions

I have an endpoint /groups
I can create a group by POSTing some info to /groups
A single group can be read by /groups/{id}
I can update some fields in the group by POSTing to /group/{id}
HOWEVER I have different fields that are needed to be updated by users with different permissions, for instance: A group might have the structure
{
"id": 1,
"name": "some name",
"members": [
{
"user_id": 456,
"known_as": "Name 1",
"user": { /* some user object */},
"status": "accepted",
"role": "admin",
"shared": "something"
},
{
"user_id": 999227,
"known_as": "Name 1",
"user": { /* some user object */},
"status": "accepted",
"role": "basic",
"shared": "something"
},
{
"user_id": 9883,
"known_as": "Name 1",
"user": { /* some user object */},
"status": "requested",
"role": "basic",
"shared": "something"
}
],
"link": "https://some-link"
}
As an example I have the following 3 operations for the /group/{id}/members/{id} endpoint:
I want only the user to be able to update his own known_as field
I want only group admins to be able to update each member's role and status fields.
I want both the user and the admin to be able to update the shared field
My options are this:
Should I allow all updates to be done by POSTing to /group/{id}/members/{id} with a subset of the fields for a member and throw an unauthorized error if they try to update a field that they aren't allowed to update?
Or should I break each operation into say /group/{id}/members/{id}/role, /group/{id}/members/{id}/shared and /group/{id}/members/{id}/status? The problem with this is that I don't want to have to make lots of requests to update all the fields (I imagine that there will end up being quite a lot of them).
So just for clarification my question is: Is it considered proper REST to do my option 1 where I can post updates to an endpoint that may fail if you try to change a field that you aren't allowed to?
In my opinion, option 1 is much better than option 2.
As you said option 2 is a waste of bandwidth.
More importantly, with option 1 you can easily implement an atomic update (update "all-or-nothing"). It should either complete successfully or fail entirely. There should never be a partial update.
With option 2 it's very likely the update can be implemented to complete some request successfully and reject another request, even if the two requests are considered a single operation.

Best Practices in Retrieving Related Data in a REST API

So I have a REST API in which I Have a resource in which other resources are linked to (related models, in programming point of view).
So how I am doing it right now is that whenever I request for a resource, related resources are referenced via URLs ('/related_data/related_data_id/').
However, I worry that, let's say there are 5 related resources to the one I'm retrieving, is that I would perform 5 GET requests. I am writing an iPhone client and I'm wondering if this is how to properly do it using REST (that I'm returning URLs). A sample JSON response is this:
{
"meta": {
"limit": 20,
"next": null,
"offset": 0,
"previous": null,
"total_count": 2
},
"objects": [
{
"away_team": "/api/team/3/",
"country": "/api/country/1/",
"event_date": "2011-08-16",
"event_time": "06:00:00",
"event_timezone": "GMT",
"home_team": "/api/team/4/",
"id": "1",
"level": "/api/level/4/",
"resource_uri": "/api/event/1/",
"tournament": "/api/tournament/1/"
},
{
"away_team": "/api/team/4/",
"country": "/api/country/1/",
"event_date": "2011-09-29",
"event_time": "12:00:00",
"event_timezone": "UTC",
"home_team": "/api/team/3/",
"id": "2",
"level": "/api/level/1/",
"resource_uri": "/api/event/2/",
"tournament": "/api/tournament/6/"
}
]
}
Is this a proper way of doing it in REST, considering that "each URI must map to a resource" and all those things?
I am using Django and django-tastypie
Thanks in advance!
Yes; that is proper if the related resources are updated independently. REST architectures depend on caching for performance, and therefore work best with resources which act as atomic entities (see more here). That way, you can update resource B and have its representation be fresh without having to update resource A. See this SO comment for more design details.