s Stefan
on

 

Dear P21,

we observed, that P21 reports generated from P21 Community 3.0.2 and 2.2.0 seem to have a limit, to the rows displayed, of 1000 within the details view for a given category of the issue summary.

I have attached two sceenshots to illustrate the problem.

e.g. according to the issue summary, we would expect to find 36657 rows for source=EG and and Pinnacle21 ID = SD1117, however only 1000 rows match the respective filter criteria within the details tab.

Is this a know limitiation of P21 Community ?

Many thanks,
Stefan

 

Forums: Troubleshooting and Problems

Sergiy
on May 18, 2020

Hi Stefan, 

This is a cool feature requested by many users of Pinnacle 21 Validator. Limitation of 1000 diagnostics records for each issue is configurable. You can manage it. 

There are two basic types of data issues. 1. Manual data entry errors 2. Programming or systematic errors. Amount of data entry errors in a study is limited and in most cases it is under 1000 per issue. So, limitation of 1000 diagnostics records per issue does not impact manual data entry errors.  A programming error is a different story.

A single programming error may introduce one or more issues affected thousands or even millions of records. In such cases, a validation report with a complete list of diagnostics records may be extremely large. Note that MS Excel cannot open spreadsheets with more than one million (1,048,575) rows. So, while validation report has valid MS Excel structure, it cannot be processed by MS Excel due to large size of report. On the other side, to fix a programming error you do not need to have millions of examples of the same data issue. It is enough to see just few records with a problem to diagnose a programming error. That why we limited a number of records in “Issue Details” tab to 10,000 per issue. Later by multiple requests from our users, we reduced this limit to 1000 records.

If you do not like such approach, you can always modify this default configuration setting.

  • In Pinnacle 21 Community, go to “Help” -> “Preferences -> “Validator”
  • In “Validator Settings” tab, modify “Excel Message Limit”. For no limit specify 0

For CLI (Command Line Interface), you can use a parameter --report.cutoff. See details at https://www.pharmasug.org/proceedings/2018/DS/PharmaSUG-2018-DS20.pdf

Please remember about MS Excel limitation. Common practice is that when our users need to see all diagnostics records, they generate a validation report in CSV format and then upload it for analysis into SAS instead of MS Excel.

Kind Regards,

Sergiy

 

 

 

s Stefan
on May 19, 2020

Dear Sergiy,

thank you for the in detail explanation. This is very helpful.

BR,
Stefan

Want a demo?

Let’s Talk.

We're eager to share and ready to listen.

Cookie Policy

Pinnacle 21 uses cookies to make our site easier for you to use. By continuing to use this website, you agree to our use of cookies. For more info visit our Privacy Policy.