I am initilizing a grid that contains more than a few thousand columns. The columns are defined by a user of my system and there is no limit to what they can define.
The performance is extremely slow. Approximately 5 minutes for the grid to render. Is there an alternative that I can do to enhance the rendering of the table?
It appears as though the columns go through a method called recursivelyCreateColumns which in my case gets called thousands of times.
Any help is appreciated
Related
My problem is simplified that:
I have several hundred of subtables within one large pyspark dataset visualized in the picture below. In that subtables there are missing values (marked in light blue) I have to calculate. I executed one operation shown in the picture. To calculate that value, I have to sum up three values to the left of the missing value and divide it through the value below. The formula is also shown in the picture.
There is no problem calculating the value in the picture, but in the next columns to the right, I have to use the results of the calculations in the columns to their left. My task is to fill out all the blue lighted cells within one subset and fill out all subsets.
My current solution is to export the pyspark dataset to pandas, creating a for loop over all columns (and subtables), read out each column (and the three to its left) within one loop to a numpy array, doing the calculation on that column and write back the result to the pandas dataframe. The algorithm is quite fast but if the number of subtables exceeds about 800 I get a memory overflow message.
Now I‘m looking for a solution in pyspark (or pandas) avoiding a loop and that out of memory proble, because the number of subtables is more than 6,000!
one subtable with formula for one missing value
What are the current limits of the data contained on a smartsheet, and what is it based on ?
I've found some links talking about 5000 records and 500 columns, but I've reached the point with a sheet containing only 1626 rows and 123 columns.
What are the exact specifications, and is there any way to override the settings using the api ?
There are maximums on the number of rows, columns, and cells in a sheet. There is no method via the Smartsheet API to override these limits. On this Smartsheet Help Center article it currently states the maximums as the following:
5,000 Rows
200 Columns
200,000 Cells
It is possible to reach one of these maximums before reaching another. 5,000 rows at 200 columns would actually be more than 200,000 cells. Once one of these maximums is reached there may be errors and the amount of data in the sheet should be decreased.
But, these aren't absolute limits. Depending on the usage of other Smartsheet features, like Formulas, Cell Linking, and Conditional Formatting, there can be limits to the amount of data that can be stored in a sheet. There are a lot of variables that can affect the performance of a sheet and the amount of data that can be stored in it, so these limits aren't absolutely specific.
This can sometimes require a process of elimination to see what can be stored in your sheet. It is best to divide up the data into logical groupings as best you can to help keep the sheets working as you need.
I have a table that will have about 3 * 10 ^ 12 lines (3 trillion), but with only 3 attributes.
In this table you will have the IDs of 2 individuals and the similarity between them (it is a number between 0 and 1 that I multiplied by 100 and put as a smallint to decrease the space).
It turns out that I need to perform, for a certain individual that I want to do the research, the summarization of these columns and returning how many individuals have up to 10% similarity, 20%, 30%. These values are fixed (every 10) until identical individuals (100%).
However, as you may know, the query will be very slow, so I thought about:
Create a new table to save summarized values
Create a VIEW to save these values.
As individuals are about 1.7 million, the search would not be so time consuming (if indexed, returns quite fast). So, what can I do?
I would like to point out that my population will be almost fixed (after the DB is fully populated, it is expected that almost no increase will be made).
A view won't help, but a materialized view sounds like it would fit the bill, if you can afford a sequential scan of the large table whenever the materialized view gets updated.
It should probably contain a row per user with a count for each percentile range.
Alternatively, you could store the aggregated data in an independent table that is updated by a trigger on the large table whenever something changes there.
I have a table with approximately 100 columns. Number of rows are not very large, about 300 rows. But it's performance is not very good.
Scrolling (especially horizontal) and adding new rows proceed with visible lags. I do not use SWT.VIRTUAL as it's good for cases when there are huge number of rows. I'm aware of alternative table implementations in SWT, but my question is particularly about SWT/JFace implementation. Is there any way to improve performance for table with quite large amount of columns?
As you already stated, SWT.VIRTUAL only helps with large number of rows. AFAIK there isn't much you can do to improve handling a large amount of columns.
... other than reducing the amoiunt of columns. Left aside the weak performance, are you sure that 100 columns will lead to a good user experience? Wouldn't it be better to choose a different form of presenting the information?
UPDATE
When I first asked this question, I didn't fully understand what the problem was. Your best bet is to glance over the issue below and then read my answer.
I have a report with a matrix where the data looks like the following:
Name Id Activity 1 Activity 2 …Acitivity N
Smith 1 77 100 nn
Johnson 2 88 99 nn
Name and Id are in a group.
When the number of activities are greater than 11, I need the columns Name and Id to repeat when the report renders in a PDF.
Here is an example of report recently run:
Page 1 (NOTE: red boxes indicate
personal data filtered out):
Page 2:
I've been fiddling with the properties RepeatRowHeaders and RepeartColumnHeaders, but I have had no success.
How do I make this column repeat?
Sadists can check out the rdl file here.
Short Answer (and a plea from me):
Please vote on this Page to conditionally hide non-group matrix columns.
Long Form:
I found the solution to my problem, but first was figuring out the exact issue. An SSRS Matrix has the ability to repeat columns on subsequent pages as its width grows when RepeatRowHeaders and RepeatColumnHeaders are set to True. So that part of my report setup was fine.
My problem was that my grid was set up as follows:
What you see above is a Tablix containing a Matrix. The blue represents a normal Tablix with four columns. The green Represents a Matrix within Column 4 that has N number of columns.
So even with the properties RepeatRowHeaders and RepeatColumnHeaders set to true, Columns 1-3 would not repeat as the data in the Matrix extended to multiple pages. This is because the outer Tablix (e.g. the blue part) only saw Column 4 as growing outwards instead of the individual columns in the inner matrix.
In order to make the columns repeat, I had to alter the grid like so:
This grid is only one matrix; not a Matrix within a Tablix as the first image depicted. I pulled the matrix out and added "Static Columns" to the left. This made Columns 1 - 3 repeat on multiple pages as the Matrix grew.
All seemed well, but here's the rub: Static Columns cannot be hidden or, more aptly put, you can hide the data within a column, but the column itself will not shrink.
So, when I hid Static Column 3, my report looked something like this:
Notice the white-space Gap. This solution simply would not work. When I found that I was stuck in an either/or situation (either have the ability to conditionally hide columns or Repeat Columns on Subsequent Pages), I used one of our Microsoft Support Incidents to analyze the report and come up with a solution.
Microsoft was nice and knowledgeable and even pointed to a article in their forum stating they were considering allowing SSRS to have the ability to hide static columns. However, in the end, they said we were stuck with the either/or situation. (BTW, note that forum thread was answered in 2006.)
So after that long process with Microsoft, we were still stuck with our dilemma. Fortunately, one of our fellow programmers came up with a workaround: put the columns you want to toggle visibility within the Matrix itself.
And that's what I did. It took some finagling with the report SQL (and creating a statement I'm not particularly proud of), but it works.
So now the working report looks like this:
That's one Matrix, two static columns that always need to repeat on every page and Column 3 which is now part of the dataset used for the Matrix.
Once again, I plead with the community to vote on this Page to conditionally hide non-group matrix columns. It would make constructing future reports like this much easier :)