I have a Database Project inside of SSDT 2012 that contains a SSIS project using the package deployment model. The goal of the project is to load a lot of information at once that would normally take too much time if one package did it. So I divided it between 15 children each doing their on separate part, loading data into various sql tables. So, inside this project is one parent package and 15 child packages. Because of the type of data that is loading, I have to use script task to insert it all. Each child package is the same, only differing between parameters that divide the data up between the children. Each child package is executed using a External Reference through the File System.
The problem I'm having is while the parent package is supposed to start all the child packages at once, not all of the children are starting. It's as if there is a limit to how many packages can start at one time (looks like about 10 or 11). Once it hits this limit, the rest don't start. But when one package finishes, another immediately starts.
Is there a property I'm missing that is limiting how packages can run at the same time? Based on what others are able to run at the same time, there seems to be something I'm missing. I read somewhere memory can be a factor, but when I look at Task Manager, I don't see anything above 15% of my memory used.
The problem is solved by looking at the the property MaxConcurrentExecutables on the parent package. In my parent package, this property had a default value of -1, which means it calculates how many tasks that run in parallel (in this case, child packages) based on the number of cores on your PC plus 2.
In my case I have 8 cores on my laptop, plus the number 2 which put me at 10 packages running at the same time. You can override this value by putting a higher positive number in its place to allow more children to run. After putting in 20, all tasks started at once.
More information about this can found here:
https://andrewbutenko.wordpress.com/2012/02/15/ssis-package-maxconcurrentexecutables/
Related
When I create a ParametersVariation simulation, the main model does not run. All I see is the default UI with iterations completed and replication. My end goal (as with most people) is to have a model go through a certain number of replications, but nothing is even running. There is limited documentation available on this. Please advise.
This is how Parameters Variation is intended to work. If you're running 1000 runs and multiple replications with parallel runs, how can you see what's happening in Main in each?
Typically, the best way to benefit from such an experiment is to track the results of each run using elements from the Analysis palette or even better to export results to Excel or similar.
To be able to collect data, you need to write your code in Java actions fields with root. to access elements in main (or top-level agent).
Check the example below, where after each run a variable from main is added to a dataset in the Parameters Variation experiment. At the end of 100 runs for example, the dataset will have 100 values of the main variable, with 1 value for each run.
I'm using mill to build a pipeline that
cleans up a bunch of CSV files (producing new files)
loads them into a database
does more work in the database (create views, etc)
runs queries to extract some files.
Should the tasks associated with steps 2 and 3 be producing something analogous to PathRef? If so, what? They aren't producing a file on the disk but nevertheless should not be repeated unless the inputs change. Similarly, tasks associated with step 3 should run if tasks in step 2 are run again.
I see in the documentation for targets that you can return a case class and that re-evaluation depends on the .hashCode of the target's return value. But I'm not sure what to do with that information.
And a related question: Does mill hash the code in each task? It seems to be doing the right thing if I change the code for one task but not others.
A (cached) task in mill is re-run, when the build file (build.sc or it's dependencies/includes) or inputs/dependencies of that task change. Whenever you construct a PathRef, a checksum of the path content is calculated and used as hashCode. This makes it possible to detect changes and only act if anything has changed.
Of course there are exceptions, e.g. input tasks (created with T.input or T.sources) and commands (created with T.command) will always run.
It is in general a good idea to return something from a task. A simple String or Int will do, e.g. to show it in the shell with mill show myTask or post-process it later. Although I think a task running something in an external database should be implemented as a command or input task (which might check, when running, if it really needs something to do), you can also implement it as cached task. But please keep in mind, that the state will only be correct if no other process/user is changing the database in between.
That aside, You could return a current database schema/data version or a last change date in step 2. Make sure, it changes whenever the database was modified. Each change of that return value would trigger step 3 and other dependent tasks. (Of course, step 3 needs to depend on step 2.) As a bonus, you could return the same (previous/old) value in case you did not change the database, which would then avoid any later work in dependent tasks.
I would like to connect 2 sources from different processes. In the second process it should pick my pallets from a pallet rack where I stored them in my first process flow.
Process 1: Store the pallets into my pallet racks.
Process 2: Pick this number of pallets to a certain time.
Can I connect this processes/ the both source blocks? Its easier for my project to seperat this process.
I know that it is possible with enter and exit but I would like to know if it is with source too
well... a source generates a completely new agent, and the enter block doesn't, so that's why you should use an enter block instead...
BUT theoretically, you could eventually remove the agent with a sink and create a new one, identical to the previous one with a source... but I don't see why you would do something like that besides a desperate attempt to save memory.... bad practice at its peak.
The real reason why you want to do this, is that you are not aware of the tools that AnyLogic has available to solve whatever you are trying to solve... maybe you are looking for a split block and separate your agent into 2 different processes? Doesn't seem so, since process 1 occurs before process 2, so i dont see why you would even want to separate them
In MDriven Enterprise Information, why is there a Processes Hierarchy and also a Processes Tree? Aren’t they the same thing? Is this not redundant duplication?
Processes are defined by their steps.
A Process step can make use of another processes - ie defining a sub-process.
To both have the complete list of processes and the resulting expanded tree of process and their sub-processes and even their sub-processes (infinity) we added two nodes.
First node is the complete straight list of existing processes regardless if they are sub-processes or not.
Second node is the constructed tree of whom uses who - and a sub-process can then show up multiple times in this tree.
Notice that ApproveNewBulk is re-used in 2 processes in the example below.
How to run two sub programs from a main program if both are db2-cobol programs?
My main program named 'Mainpgm1', which is calling my subprograms named 'subpgm1' and 'subpgm2' which are a called programs and I preferred static call only.
Actually, I am now using a statement called package instead of a plan and one member, both in 'db2bind'(bind program) along with one dbrmlib which is having a dsn name.
The main problem is that What are the changes affected in 'db2bind' while I am binding both the db2-cobol programs.
Similarly, in the 'DB2RUN'(run program) too.
Each program (or subprogram) that contains SQL needs to be pre-processed to create a DBRM. The DBRM is then bound into
a PLAN that is accessed by a LOAD module at run time to obtain the correct DB/2 access
paths for the SQL statements it contains.
You have gone from having all of your SQL in one program to several sub-programs. The basic process
remains the same - you need a PLAN to run the program.
DBA's often suggest that if you have several sub-programs containing SQL that you
create PACKAGES for them and then bind the PACKAGES into a PLAN. What was once a one
step process is now two:
Bind DBRM into a PACKAGE
Bind PACKAGES into a PLAN
What is the big deal with PACKAGES?
Suppose you have 50 sub-programs containing SQL. If you create
a DBRM for each of them and then bind all 50 into a PLAN, as a single operation, it is going
to take a lot of resources to build the PLAN because every SQL statement in every program needs
to be analyzed and access paths created for them. This isn't so bad when all 50 sub-programs are new
or have been changed. However, if you have a relatively stable system and want to change 1 sub-program you
end up reBINDing all 50 DBRMS to create the PLAN - even though 49 of the 50 have not changed and
will end up using exactly the same access paths. This isn't a very good apporach. It is analagous to compiling
all 50 sub-programs every time you make a change to any one of them.
However, if you create a PACKAGE for each sub-program, the PACKAGE is what takes the real work to build.
It contains all the access paths for its associated DBRM. Now if you change just 1 sub-program you
only have to rebuild its PACKAGE by rebinding a single DBRM into the PACKAGE collection and then reBIND the PLAN.
However, binding a set of PACKAGES (collection) into a PLAN
is a whole lot less resource intensive than binding all the DBRM's in the system.
Once you have a PLAN containing all of the access paths used in your program, just use it. It doesn't matter
if the SQL being executed is from subprogram1 or subprogram2. As long as you have associated the PLAN
to the LOAD that is being run it should all work out.
Every installation has its own naming conventions and standards for setting up PACKAGES, COLLECTIONS and
PLANS. You should review these with your Data Base Administrator before going much further.
Here is some background information concerning program preperation in a DB/2 environment:
Developing your Application