I need to connect Salesforce to an external database we have, and constantly keep both the database and salesforce updated in as close to real time as we can get. I have tired Google searching possible solutions, but nearly all of them have been outdated by over a year. Any ideas?
Thank You!
Depending on your exact scenario it is quite difficult to give you a proper answer.
However off the top of my head I would suggest two Salesforce products.
Salesforce Connect
https://www.salesforce.com/products/platform/products/salesforce-connect/
Salesforce Connect allows you to connect to various data sources and turn the tables / objects of that data source into a SObject. For example MySQL, Microsoft SQL Server, Oracle etc. There are limitations and thus it would be better to talk to a Certified Architect about such an implementation.
Heroku Connect
https://www.heroku.com/connect
Heroku Connect allows you to connect a Heroku data source with a Salesforce Object. The sync is not immediate but there are quite a few customisations inside the product to make the sync as "live" as possible. There are limitations and thus it would be better to talk to a Certified Architect about such an implementation.
Salesforce Connect has limitations.. It's good for presenting data via the interface, but if you need to act on the data and report on the data it might not be the best bet.
For close to real time hand coded sync, look at the streaming API, or using Salesforce Platform Events.
If you want to use an ETL tool, my organization has had decent luck with DBAmp, which is a Sql add on product and fairly inexpensive as compared to a lot of ETL tools ($1625 annually.) http://www.forceamp.com/ We're able to replicate the entire SF database offline in SQL with DBAMP, push changes to the offline Sql copy and upsert changes. It's also a good backup solution via offline full data copy. We got very good support from them as well when we encountered challenges.
Hope this helps.
Not sure if you are syncing one object or multiple objects but there are a few options that you have.
You can try the salesforce provided features Salesforce Connect which allows you to view and update data from your external source In salesforce but there are limitations with reporting and other considerations you should consider.
If you make use of Heroku, Heroku Connect is your best bet
You can also use a middleware ESB solutions like MuleSoft which can orchestrate keeping data in sync across multiple data sources and do batch loads, but depending on how often changes you want to keep an eye out for api limits for inbound calls to salesforce.
You can roll your own solution where you can use Outbound Messages in workflow (or triggers that initiates an apex class that calls out, but that is more cumbersome and you have to do custom error handling and retry logic which you get for free using outbound messages) to send changes from salesforce to your homegrown service that writes to you database and have you homegrown solution write back to salesforce using the soap or rest api. That would probably take you some time to build. You would also still need to be aware of API limits depending on how many updates are made on the non salesforce side.
You crate a Canvas App which displays data from your DB in Salesforce as a Tab and hook it up via SSO so users are auto logged in. But again there would not be reporting, or any salesforce features that you can take advantage of.
But I really think that you should spend some time to determine what system is your source of truth because that would determine how the data should be synced. You should also investigate if you really need the sync to be realtime or near realtime, or if you can manage with something like an hourly true up on the system that is not the source of truth.
Related
We'd like to allow our users to make changes to our ERP (only very limited conditions) from within the smartsheet. Is there a way to add a web service call based on changes to a cell?
It's certainly possible to do what you've described (provided, of course, that your ERP supports inbound web service calls to update data there).
If you're wanting to do this programmatically (i.e., by writing a script in Python or any other supported language), you can use the Smartsheet API to create one or more webhooks that will monitor Smartsheet for the type of changes you specify, and send your integration notifications when those events occur. Your integration would then listen for those inbound event notifications, and when they are received, it would programmatically issue the appropriate web service call(s) to update data in the ERP.
It's also worth mentioning that, depending on what ERP you're using, it may be possible for you to accomplish your goal without having to write any code. If you're interested in exploring the feasibility of that approach, I'd suggest you check out products like Zapier, Power Automate, etc. to see if they offer a connector for Smartsheet and your ERP. You may also want to check out the Smartsheet Connectors and Integrations page to see if there's a connector for your ERP listed there.
Background:
I'm building an API service app. The app is just like any other, you send an HTTP request and receive a response. This seems simple up until I start thinking about user registration, payments, authentication, logging and so on.
Application:
tl;dr simple app diagram
Endpoints listening for HTTP requests and doing all the request related work. This is the core of the service, what the service user would use this app for. Directly not accessible to the end user (unless somehow it knows the url). Python flask server, deployed on google cloud RUN.
API gateway acting like a proxy and a single access point forwarding the requests to the endpoints. This is the service access point for the end users. This part will also be responsible for authentication, limitations, logging and tracking the use of the API endpoints. Python flask server, deployed on google cloud RUN.
Website including documentation, demo and show off of API calls through API gateway, registration, payment (thinking of Stripe) etc. VueJS app on NodeJS server on google cloud compute VM.
Database storing credentials of registered users, payment information and auth keys. Not implemented yet.
Problems:
Is this architecture proper? What could be done differently or improved? How could I further simplify all the interactions between separate parts of the app? Am I not missing any essential parts?
Haven't yet implemented the database part and I'm not sure what should I
use? There are plenty of options on google cloud. Also I could go with something simple and just install a DB with http/JSON interface on google cloud compute VM. How do I chose the DB? Given such an app, what would be the best choice?
Please recommend literature/blogs/other sources of info on similar app
architecture for new developers not familiar with it?
This is pretty open ended, but here are some general comments:
Think about how your UI will work. Are you setting up a static app served directly from cloud storage or do you need something rendered on the server? Personally I prefer separating UI from API when I can but you need to be aware of things like search engine optimization. Even if you need to render some content dynamically your site can still be static. Take a look at static site generators like Gatsby. I haven't had to implement a server rendered UI in years and that makes me happy.
API gateway might be fine, but you don't really need it for anything. It might be simpler to start without it and concentrate on what actually matters. If your APIs are being called by an external client you can't trust the calls anyways and any API key you might be using will be exposed. I'd say don't worry about it for a single app. That being said, if you definitely want to use a GW then use one, just be aware that it is mostly a glorified proxy and not some core part of your architecture.
Make sure your API implementations don't store any local state so you can rely on Cloud Run scaling your services up and down. Definitely don't ever store state directly inside your containers. If you need state on the server it needs to be in some external data store.
Use JWTs or an external IDM (that will generate JWTs) for authentication. Keep session data on the client side as much as possible and pass the JWT in every API call to authenticate the caller. If you are implementing login on your own the only APIs you need to expose without tokens are for auth and password recovery, which you can separate into their own service.
Database selection depends on how well you understand your processes, how transactional your services are and your existing skillset. Overall I would use what you are comfortable with, you can probably succeed with a lot of things. Certain NoSQL flavors can seem simple on the surface but if you don't have a clear understanding on the types of queries you need to run they can get tedious to work with. Generally you should stick to relational databases for OLAP style implementations and consider NoSQL for OLTP. Personally I like MongoDB and it is very popular, probably because it sort of sits in the middle of the pack which makes it fit a lot of applications. Using MongoDB also makes you cloud agnostic since it is available on every platform. Using platform specific database flavors can lock you down to a specific vendor.
Whatever you do, don't start installing things on VMs. You can be almost 100% sure you are doing it wrong if this comes up. Remember, the services you consume don't all have to be managed by Google or even run on GCP. You can get MongoDB capacity directly from MongoDB who manage it on your behalf on all of the Big3 cloud vendors.
At least think about the long term, even if you don't necessarily need to have it impact your architecture right now. If you are expecting your app to be up for years try to make it more platform agnostic than less. This might mean sticking away from some really platform specific serverless features that will force you to jump a couple of extra hoops. If you are using Cloud Run you are using containers which already makes your app pretty portable, don't lock it to one platform by using a lot of platform specific features. That being said, don't stay away from them either. You should always go for the low hanging fruit, so don't try to avoid using things like secrets manager etc. If your app has a short lifespan and you need really fast time to market then don't worry about it.
Just my 2c, what you are doing is very generic and can be done in a lot of different ways.
I have some tables from Postgres database to be integrated into Salesforce as external objects. I went through some video tutorials and documentations where I was recommended to use Salesforce Connect which supports providers with "OData" protocol support. Is it possible to integrate Postgres tables into Salesforce as external objects without Salesforce Connect?
Thanks.
Be careful with the phrase "external objects". To me, the use of those particular words implies the specific implementation of external data access/federation delivered with Salesforce Connect. I don't believe that there is any alternative if your goal is to create "real" external objects (named "objectname__x") within Salesforce.
There are, though, Salesforce integration solutions from the likes of Progress, Jitterbit, Mulesoft, and Informatica and others that can be used to access PostgreSQL, with varying degrees of coding being required. You won't get "external objects", but you will be able to access data residing off-cloud in a PostgreSQL database from your Salesforce system.
Hope this helps.
Currently the way to integrate data from external storages (Postgres in your case) without Salesforce Connect is implement your custom logic for synchronization using REST or SOAP API, Apex classes and triggers, Salesforce Workflows and Flows. Also you will need to implement appropriate interfaces on side of your data storage. Complexity of all these steps depends on complexity of your existing data model and infrastructure around it.
A question regarding the integration of the document management system Alfresco into Oracle Application Express (APEX) based on CMIs-repository:
The aim is to use APEX as the portal-page and Alfresco showing it's results (document lists) based on search parameters coming form APEX.
A search result from a CMIS-query should be displayed in an APEX page-region.
Unfortunately I have no experience in this sector (REST, CMIS) - so any advice would be welcome!
A related question regarding user authentication and authorization via CMIS does also arise.
Has anyone out there implemented something like this or used these components together, yet?
The first thing that pops into my mind is making the choice where you want your communication with the repository to take place: client side or server side?
Alfresco supports Web Scripts, so I would be possible to create a javascript-heavy thick client which connects to your repository, get information about your files and redirect to their download links.
The alternative would be to design some way to connect to the repository from the database server. Again there are many ways to do this. You can connect to the repository during your page load and use PL/SQL regions to fire scripts that connect to your repository, get the data you want, and render your region with that information.
Another way would be to periodically check the repository for changes, and maintain a 'shadow copy' of the repository within your oracle database tables.
Of course all of these solutions have their own drawbacks.
Okay, so I'm currently developing an iphone app that I plan to take into production and scale. I'm a bit lost on the whole subject.
What is better to use: core data or sqlite? (as the local DB)
Also, can sqlite be used exclusively to communicate with my remote server as well? At first I thought it could but I've been reading that sqlite isn't great to use on servers that get a massive amount of hits.
I've read that oracle, mysql, or mssql may be better to use on a remote server and that I can communicate with these servers via REST or SOAP.
I plan to be able to both read and write to a remote server. The files transferred will mostly be small data objects and pictures. Speed is of the essence, so I'd like to know which options are my fastest routes. Of course, I want the option to scale and not have performance take too much of a hit as well.
On the subject of Core Data vs sqlite see this question.
SQLite is a small and lite embedded SQL database engine. It's not meant to used in server environments. In general, it's not a good idea to communicate directly over the Internet. It's more common to have some sort of process logic between the client code and the database to do a range of things like validate input, process business logic, security, etc. You can implement this sort of layer in REST, SOAP, or whatever you like. Since your clients will be mobile devices, a http based web service (like REST or SOAP) is a good idea as all mobility platforms have inbuilt API support for http messaging. There are lots and lots of options on the server depending on what type of server you want to setup and run with.
If your new to this, maybe you should read something like 'Patterns of Enterprise Application Architecture' by Martin Fowler to get a idea of what sort of design patterns people use to implement the server side layering.