Our customer has an existing store based on FEP4/Madisons migrated all the way from several major versions ago WCS. They want to transition to a new store based on Aurora. The logic project and front-end interaction with it is not up to date at all, so along the way we want to migrate that.
We are trying to make a rough plan for the project.
The other guys in the company (who are unavailable right now - great!) have told me that their experience is that while a FEP upgrade should only take about a week, it always end up taking at least a month.
I am trying to establish if there is a big time difference between these two scenarios:
Scenario 1: "I want to upgrade an existing Madisons-based store and the associated WCS infrastructure and DB2 schema/data from FEP4 to FEP6" In this world we keep the existing development images and the various other servers along from dev to PROD, and upgrade the existing store + infrastructure to FEP6 before we start building the new store.
Scenario 2: "I want to create an entirely new store based on Aurora and the existing data for the old store, and upgrade the associated WCS infrastructure and DB2 schema/data from FEP4 to FEP6". In this world we create all new environments for dev ... prod, and so have a FEP4 branch for the current store, and a FEP6 branch for the new store.
Can you perhaps give a rough estimate of how long to would take, and how the time will be (roughly) distributed over these four headlines A very rough estimate is fine...
Upgrade/install WCS components : _ % of time
Upgrade/migrate DB2 schema and data: _ % of time
Upgrade and fix front end issues: _ % of time
"Other tasks": _ %
Upgrade/install WCS components : 1.5 day
Upgrade/migrate DB2 schema and data: % of time (no need to migrate DB2 schema) just publish Aurora and copy Aurora store JSP's to your Madisons Store. you should do Madisons to Aurora migration .
Upgrade and fix front end issues: _ % of time it's [depend on your customization.]
Thanks
Related
I'm building an application where each of our clients needs their own data warehouse (for security, compliance, and maintainability reasons). For each client we pull in data from multiple third party integrations and then merge them into a unified view, which we use to perform analytics and report metrics for the data across those integrations. These transformations and all relevant schemas are the same for all clients. We would need this to scale to 1000s of clients.
From what I gather dbt is designed so each project corresponds with one warehouse. I see two options:
Use one project and create a separate environment target for each client (and maybe a single dev environment). Given that environments aren't designed for this, are there any catches to this? Will scheduling, orchestrating, or querying the outputs be painful or unscalable for some reason?
profiles.yml:
example_project:
target: dev
outputs:
dev:
type: redshift
...
client_1:
type: redshift
...
client_2:
type: redshift
...
...
Create multiple projects, and create a shared dbt package containing most of the logic. This seems very unwieldy needing to maintain a separate repo for each client and less developer friendly.
profiles.yml:
client_1_project:
target: dev
outputs:
client_1:
type: redshift
...
client_2_project:
target: dev
outputs:
client_2:
type: redshift
...
Thoughts?
I think you captured both options.
If you have a single database connection, and your client data is logically separated in that connection, I would definitely pick #2 (one package, many client projects) over #1. Some reasons:
Selecting data from a different source (within a single connection), depending on the target, is a bit hacky, and wouldn't scale well for 1000's of clients.
The developer experience for packages isn't so bad. You will want a developer data source, but depending on your business you could maybe get away with using one client's data (or an anonymized version of that). It will be good to keep this developer environment logically separate from any individual client's implementation, and packages allow you to do that.
I would consider generating the client projects programmatically, probably using a Python CLI to set up, dbt run, and tear down the required files for each client project (I'm assuming you're not going to use dbt Cloud and have another orchestrator or compute environment that you control). It's easy to write YAML from Python with pyyaml (each file is just a dict), and your individual projects probably only need separate profiles.yml, sources.yml, and (maybe) dbt_project.yml files. I wouldn't check these generated files for each client into source control -- just check in the script and generate the files you need with each invocation of dbt.
On the other hand, if your clients each have their own physical database with separate connections and credentials, and those databases are absolutely identical, you could get away with #1 (one project, many profiles). The "hardest" parts of that approach would likely be managing secrets and generating/maintaining a list of targets that you could iterate over (ideally in a parallel fashion).
I am a beginner in anylogic and I am modeling production process in which there are several different raw materials (defined as agents e.g. metal sheet A, metal sheet B, etc.) they are going through the process and create several types of final products… therefor I have to inject the materials based on production plan and store them in warehouse and then get them out of the warehouse according to the production plan and BOM of the product.
Let’s say that the situation is that Any logic should check the production plan hourly and inject the agents to the model according to that production plan. Here is a part of injection plan according to my monthly production plan. (I have this plan for 1 month and I want to simulate it.)
My question is that how can I set this production plan for anylogic to inject the materials to the model.
Good questions. This is typically where I would suggest making use the AnyLogic database and using a Dynamic event to inject the right amount of units at the time specified.
I start by setting up an Excel sheet with the schedule data and importing it into the AnyLogic DB
Then you need a Dynamic event with code similar to the one below that will inject the correct amount of units at the right time for the right source.
And then on the model start-up you call a function that creates all these dynamic events for you from the entries in the DB
Should be very simple and this approach is very flexible able to handle any schedule and any amount of units
I am using Entity Framework and I have a table that I use to record some events that gets generated by a 3rd party. These events are valid for 3 days. So I like to clean out any events that is older than 3 days to keep my database table leaner. Is there anyway I can do this? Some approach that won't cause any performance issue during clean up.
To do the mentioned above there are some options :
1) Define stored procedure mapped to your EF . And you can use Quarts Trigger to execute this method using timely manner.
https://www.quartz-scheduler.net
2) SQL Server Agent job which runs every day at the least peak time and remove your rows.
https://learn.microsoft.com/en-us/sql/ssms/agent/schedule-a-job?view=sql-server-2017
If only the method required for cleaning purpose nothing else i recommend you go with option 2
first of all. Make sure the 3rd party write a timestamp on every record, in that way you will be able to track how old the record is.
then. Create a script that deletes all record that are older than 3 days.
DELETE FROM yourTable WHERE DATEDIFF(day,getdate(),thatColumn) < -3
now create a scheduled task in SQL management Studio:
In SQL Management Studio, navigate to the server, then expand the SQL Server Agent item, and finally the Jobs folder to view, edit, add scheduled jobs.
set script to run once every day or whatever please you :)
i have a sqlite file in my app. the app is with enterprise certificate .
Now , if i am going to create new version with the changes in database then :
1) i have to change my Sqlite file name ?
2) if yes , then is there any way that if any person who is using my app, who has his saved data in app , will be inserted automatically in new version of my app .
otherwise if i create new file and then the older data would be removed!!!
Thanks in advance..
1) AFAIK, when you update, the old app bundle is replaced with the new app bundle. But the user data is untouched. i.e. the copy of the sqlite DB which the app changes with use by the user, is not deleted/replaced.
2) So, you can possibly ship changes to the DB as queries:
a) New tables go in as CREATE TABLE queries.
b) Changes to table structure go in as ALTER TABLE queries.
3) Now, one problem is, you didn't think about all this when you shipped the older version. So how will you make sure the queries 2a and 2b are executed only on upgrade?
a) When you first get control in the application, before allowing user interaction, fire a dummy SELECT SINGLE * FROM <tablename> on a table you have 'changed' in new version.
b) Use sqlite3_column_count and sqlite3_column_name to find out which version of the app the DB corresponds to [old or new].
c) If it is new version (i.e. new columns already exist), do nothing, else fire the queries I mentioned earlier as 2a and 2b.
If you haven't started by using this, then it's probably moot, but Core Data handles migrations (IMHO) fairly well; enough that it's worth the hurdle of learning the API. We've had great success with it (including complex migrations, inserting new objects "between" relationships, etc).
Windows Workflow Foundation has a problem that is slow when doing WF instances persistace.
I'm planning to do a project whose bussiness layer will be based on WF exposed WCF services. The project will have 20,000 new workflow instances created each month, each instance could take up to 2 months to finish.
What I was lead to belive that given WF slownes when doing peristance my given problem would be unattainable given performance reasons.
I have the following questions:
Is this true? Will my performance be crap with that load(given WF persitance speed limitations)
How can I solve the problem?
We currently have two possible solutions:
1. Each new buisiness process request(e.g. Give me a new drivers license) will be a new WF instance, and the number of persistance operations will be limited by forwarding all status request operations to saved state values in a separate database.
2. Have only a small amount of Workflow Instances up at any give time, without any persistance ofso ever(only in case of system crashes etc.), by breaking each workflow stap in to a separate worklof and that workflow handling each business process request instance in the system that is at that current step(e.g. I'm submitting my driver license reques form, which is step one... we have 100 cases of that, and my step one workflow will handle every case simultaneusly).
I'm very insterested in solution for that problem. If you want to discuss that problem pleas be free to mail me at nstjelja#gmail.com
The number of hydrated executing wokflows will be determined by environmental factors memory server through put etc. Persistence issue really only come into play if you are loading and unloading workflows all the time aka real(ish) time in that case workflow may not be the best solution.
In my current project we also use WF with persistence. We don't have quite the same volume (perhaps ~2000 instances/month), and they are usually not as long to complete (they are normally done within 5 minutes, in some cases a few days). We did decide to split up the main workflow in two parts, where the normal waiting state would be. I can't say that I have noticed any performance difference in the system due to this, but it did simplify it, since our system sometimes had problems matching incoming signals to the correct workflow instance (that was an issue in our code; not in WF).
I think that if I were to start a new project based on WF I would rather go for smaller workflows that are invoked in sequence, than to have big workflows handling the full process.
To be honest I am still investigating the performance characteristics of workflow foundation.
However if it helps, I have heard the WF team have made many performance improvements with the new release of WF 4.
Here are a couple of links that might help (if you havn't seem them already)
A Developer's Introduction to Windows Workflow Foundation (WF) in .NET 4 (discusses performance improvements)
Performance Characteristics of Windows Workflow Foundation (applies to WF 3.0)
WF on 3.5 had a performance problem. WF4 does not - 20000 WF instances per month is nothing. If you were talking per minute I'd be worried.