I have long variable names and a questionnaire with ~9 dimensions. The summary scores are therefore impossible to read.
How can I export the summary results from my mirt analysis into a proper table in excel or other format?
The methods I use for exporting other results do not seem to work with the mirt summary, and neither do basic dataset export methods. So far I have only been able to make readable tables by hand (which takes forever).
Related
Dear Anylogic Community,
I am struggling with finding the right approach for storing my simulation results. I have datasets created that keep track of every value I am interested in. They live in Main (see below)
My aim is to do a parameter variation experiment. In every run, I change the value for p_nDrones (see below)
After the experiment, I would like to store all the datasets in one excel sheet.
However, when I do the parameter variation experiment and afterwards check the log of the dataset (datasets_log), the changed values do not even show up (2 is the value I did set up in the normal simulation).
Now my question. Do I need to create another type of dataset if I want to track the values that are produced in the experiments? Why are they not stored after executing the experiment?
I really would appreciate if someone could share the best way to set up this export of experiment results. I would like to store the whole time series for every dataset.
Thank you!
Best option would be to write the outputs to some external file at the end of each model run.
If you want to use Excel, which I personally would not advise, even though it has a nice excelFile.writeDataSet() function, you can.
I would rather write the data to a text file as you will have much for control over the writing, the file itself, it is thread-safe, and useable in many many more platforms than Microsoft Excel.
See my example below:
Setup parameters in your model that you will write the data to at the end of the model of type TextFile. Here I used the model on destroy code to write out the data from the data sets.
Here you can immediately see the benefit of using the text file! You can add the number of drones we are simulating (or scenario name or any other parameter) in a column, whereas with Excel this would be a pain...
Now you can pass your specific text file to the model to use by adding it to the parameter variation page, providing it to the model through the parameters.
You will see that I also set up some headers for the text file in the Initial Experiment setup part, and then at the very end of the experiment, I close the text files in the After experiment section so that the text files can be used.
Here is the result if you simply right-click on the text files and open them in Excel. (Excel will always have a purpose, even if it is just to open text files ;-) )
I am trying to use parameter variation in AnyLogic. My inputs are 3 parameters, each varying 5 times. My output is water demand. What I need from parameter variation is the way in which demand changes according to the different combinations of the three parameters. I imagine something like: there are 10,950 rows (one for each day), the first column is time (in days), the second column are the values for the first combination, the second column is the second combination, and so on and so forth. What would be the best way to track this metadata to then be able to export it to excel? I have added a "dataset" to my main to track demand through each simulation, but I am not sure what to add to the parameter variation experiment interface to track the output across the different iterations. It would also be helpful to have a way to know which combination of inputs produced a given output (for example, have the combination be the name for each column). I see that there are Java Actions, but I haven't been able to figure out the code to do what I need. I appreciate any help with this matter.
The easiest approach is just to track this in output database tables which are then exported to Excel at the end of your run. As long as these tables include outputs from multiple runs (and are, for example, only cleared at the start of the experiment not the run), your Parameter Variation experiment will end up with an Excel file having outcomes from all the runs. (You will probably need to turn off parallel execution in the PV experiment so you don't run into issues trying to write to the same Excel file in parallel.)
So, for example, you might have tables:
run_details with columns id, parm1, parm2 and parm3 (with proper column names given your actual parameters and some unique ID generated for each run)
output_demand with columns run_id, sim_time_hrs and demand_value (if, say, you're storing some demand value each hour of simulated time) where run_id cross-references the run's ID in run_details
(There is extra complexity in how you could allocate a unique run ID and how and when you write to/clear those tables, but I'm just presenting the core design. You can also get round the need-serial-execution point by programmatically controlling when you export to Excel, rather than using the built-in "Export tables at the end of model execution" capability, but that's also more complicated.)
Tableau is an excellent tool for visualizing data. However, it is designed to be the final stop in a data (ETL) pipeline.
My Tableau workbook uses a bunch of Table Calcs to generate a list of "recommended orders". Rather than view these, I want to automate and execute them. This would make Tableau the engine of a quasi-ML process.
In other words, I would like to make Tableau a part of my ETL pipeline and send data to another tier. How can I write a back-end program that executes my Tableau workbook and receives a results dataset?
See the end of this article for example data I want to automate:
http://robm26.blogspot.com/2015/10/keep-your-factory-humming-with-tableau.html
Any ideas?
You're not not going to like the answer I'm going to give you -- "Don't do this".
Tableau isn't meant to be a task in a larger ETL pipeline and the reason you're having problems making it behave the way you want is it's not meant to be done.
Above and beyond the fact that you've figured out how to get a result that you want in Tableau ("the work is done"), Tableau isn't offering you any real value in the scenario you're describing. Use a tool (like Alteryx) that is really purpose built for this sort of work.
The above answer is correct that tabcmd is the way to pull it out. We use a function in python to generate the tabcmd requests so that they can be batched.
import subprocess
def runTabCmd(cmd):
# run tableau command and display the output
print cmd
if run_tabcmd == 'yes':
p = subprocess.Popen(
cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
print line
You probably already knew that, but for us it was a way to completely automate the pulling and loading into another python package like scikit-learn for a streamlined ML solution
I'm editing this answer to agree with Russell's answer. Tableau is not an ETL tool and should not be used as such. If you absolutely have to do something, you can use what I provided. Otherwise, the best practice is to use a tool designed for the job.
You can easily use tabcmd to get the results of a view in CSV, which can be used later in your ETL process. If you need to automate it, you can write a script and execute it with a cron job. I, myself, have a few views that are exported to CSV and used later in my ETL stream to feed our CRM.
Just remember to create the view exactly as you want it to be exported to CSV - usually including the order of the fields. Another tip is that I don't let it use the default "Measure Names" and "Measure Values" - to make sure everything is good on my CSV, I have the fields added manually in the row/columns section.
I'm using Access 2010 and I would like to create a form where useres can enter data much like they can in an Excel format. The users need to enter a date when a step is complete for a specific unit.
In Excel the units run aross the top row, the steps run down the left column. In the cell where the unit and step meet, you put a date when it is done. In Access it is much harder to create a form that looks like this (at least it has been for me so far). I tried to use a crosstab query, but you cannot enter information into a crosstab query. I can do a massive form listing every combination of units and steps, but there are over 50 units and each one has 63 steps. I don't want users to have to look through a form with over 3,000 lines in it just to enter a date completed.
This definitely feels like something we should be using a database for instead of an Excel spreadsheet. I have the format for the back end tables to hold the data. My problem is how to create a form that is easy for the user to enter the dates.
I think you have three options. First, you could build the form from scratch, which you've already admitted would suck. And it would take a lot of VBA to get the data in the right place. Second, you could automate Excel - it's the same as the first method but you don't have to build the form. You would populate an Excel spreadsheet and write the changes back to the database. This is not a trivial task and I don't recommend it.
The third option is to throw away the idea that you want to do this in a grid. You've probably been doing it in Excel and to the extent that you replicate that, your users will be happy and comfortable. Moving to a database gives you benefits, but there are costs. This is one of the costs.
So create a form with a dropdown containing all of the units. When a unit is selected, populate a subform (datasheet view) listing all of the procedures and dates. The user can then select the unit, and fill in the date next to whichever procedures he wants. Then he can select another unit and do the same thing. If you set it up right, your data will flow to the tables automatically and you'll get all the benefits of data validation, input masks, relationships, and whatever else you're using at the table level.
I know that's not what you were looking for, but I think it's the best way forward.
One of the features in my application is a log where a user can add log entries. I want to make it possible to for the user to export this data. However I do not know which format I should use for this. The data looks like this:
A date, distance, duration, maximum four category names. What I want is to make it possible to send it on mail or open it with dropbox using the URL scheme if the user has dropbox.
I have read about CSV format but I don't know if that is a good file format? My main concern is that the user do not have to have a fixed number of categories (could be between 1-4 categories)
Seeing as the columns of data to be exported will be dynamic in total, it will depend on what the user selects - and there's nothing wrong with this.
I think .csv is fine for this purpose as well - but you need to ask yourself... what will the user be doing with the data? You could either offer multiple file export formats or whatever is the best-for-purpose format, depending on what your average user will do with it.
CSV (comma separated values) is simple (and adds very little overhead - the commas), but not terribly flexible. This is good for importing to MSFT Excel, for instance.
You should consider using XML (the same underlying format used for plists) which is a very flexible (future proof should you wish to add additional columns in the future) and well supported format.