The increased utilization of cells in biomanufacturing and as therapeutic products over the last decade has prompted the development and publication of two ISO Cell Counting Standards, ISO 20391 – 1:2018 and ISO 20391 – 2:2019 to provide guidance on general principles relating to cell counting and to establish an approach to evaluate the quality of cell counting methods. In this work, we demonstrate the practical implementation of the experimental protocol outlined in ISO Cell Counting Standard Part 2 and a BlandAltman comparative analysis to evaluate performance and comparison of cell counting methods. We compare two cell types, two image cytometry instruments, and two fluorescent stains, calculating the precision, coefficient of determination (R^{2}), and a proportionality index (PI) parameter to evaluate cell counting method performance. In addition, the cell counting results are directly compared to evaluate bias between two cell counting methods. The protocol is suitable for evaluating and comparing the performance of multiple cell counting methods to select for downstream assays.
In the recent decade, cell and gene therapies have drastically improved their efficacy and have become essential players in cancer treatment [1]Boyiadzis MM, Dhodapkar MV, Brentjens RJ et al. Chimeric antigen receptor (CAR) T therapies for the treatment of hematologic malignancies: clinical perspective and significance. J. ImmunoTher. Cancer 2018; 6(137): 1–12., [2]Gonçalves GAR, Paiva RdMA. Gene therapy: advances, challenges and perspectives. Einstein 2017; 15(3): 369–75.. With the approval of two chimeric antigen receptor (CAR) T cell therapies by the U.S. Food and Drug Administration (FDA) in 2017, the numbers of clinical studies and tests on new and novel cell therapy products have also surged [3]Huang R, Li X, He Y et al. Recent advances in CART cell engineering. J. Hematol. Oncol. 2020; 13(86): 1–19., [4]Li Y, Huo Y, Yu L, Wang J. Quality Control and Nonclinical Research on CART Cell Products: General Principles and Key Issues. Engineering 2019; 5: 122–31., [5]Seimetz D, Heller K, Richter J. Approval of First CARTs: Have we Solved all Hurdles for ATMPs? Cell Med. 2019; 11: 1–16.. Typically, cellular therapies require genetic modification of the immune cells (i.e. T cells, NK cells) collected from patients, culture expansion, and reintroduction of the final products back into the patients. Therefore, it is critical to provide accurate cell counting for the administration of proper dosages, which may otherwise lead to inefficacy or induce unwanted autoimmune responses in patients undergoing therapeutic treatments [6]Nagata S, Hanayama R, Kawane K. Autoimmunity and the Clearance of Dead Cells. Cell 2010; 140(5): 619–30., [7]Rock KL, Kono H. The inflammatory response to cell death. Ann. Rev. Pathol. 2008; 3: 99–126., [8]Zhang Y, Chen X, Gueydan C, Han J. Plasma membrane changes during programmed cell deaths. Cell Res. 2018; 28: 9–21..
In the 21st Century Cures Act, the United States Congress has also recognized the importance of standardization for streamlining development, quality assurance, and facilitating regulatory approval of cell and gene therapy products [9]. In the “Synergizing Efforts in Standards Development for Cellular Therapies and Regenerative Medicine Products” workshop held by the FDA on March 31st, 2014, cell counting and viability measurement assurance were identified as opportunities for standards development [10]Arcidiacono JA, Bauer SR, Kaplan DS, Allocca CM, Sarkar S, LinGibson S. FDA and NIST collaboration on standards development activities supporting innovation and translation of regenerative medicine products. Cytotherapy 2018; 20: 779–84., [11]LinGibson S, Sarkar S, Elliott JT. Summary of the National Institute of Standards and Technology and US Food And Drug Administration cell counting workshop: Sharing practices in cell counting measurements. Cytotherapy 2018; 20(6): 785–95.. ISO has since published two cell counting standards, “ISO 203911:2018 Biotechnology – Cell Counting – Part 1: General Guidance on Cell Counting Methods” and “ISO 203912:2019 Biotechnology – Cell Counting – Part 2: Experimental Design and Statistical Analysis to Quantify Counting Method Performance”, which can serve as guidance for researchers working in the field of immunotherapy and adoptive cell therapy, where both require high quality and robust cell counting measurements for biologics and cell products [12]Biotechnology – Cell counting – Part 1: General guidance on cell counting methods. In: Standardization IOf, editor; 2018., [13]Biotechnology – Cell counting – Part 2: Experimental design and statistical analysis to quantify counting method performance. In: Standardization IOf, editor; 2019..
Derived from general concepts described in ISO Cell Counting Standard Part 1, we propose 6 key factors that can provide guidance on the selection of cell counting methods and improve the quality of the cell counting measurements:
It is also realized that the cell counting needs for cell and gene therapies are broad due to a wide range of biological sample types with various formulations and bioprocessing steps, which are complex, dynamic, and heterogeneous. Because there are currently no reference materials for live mammalian cells that are certified for cell concentration, the accuracy parameter outlined in the ICH Harmonised Tripartite Guideline – Validation of Analytical Procedures: Text and Methodology Q2 (R1) cannot be readily applied, thus increasing the challenge and difficulty of validating the accuracy of cell counting [14]Validation of Analytical Procedures: Text and Methodology Q2(R1). In: Guideline IHT, editor; 2005., [15]Guidance for Industry: Process Validation: General Principles and Practices. In: Services USDoHaH, Administration FaD, (CDER) CfDEaR, (CBER) CfBEaR, (CVM) CfVM, editors; 2011., [16]Guidance for Industry: Analytical Procedures and Methods Validation for Drugs and Biologics. In: Services USDoHaH, Administration FaD, (CDER) CfDEaR, (CBER) CfBEaR, editors; 2015.. Therefore, the ISO cell counting standards can serve as a valuable tool to evaluate and select cell counting methods that are fitforpurpose, in order to increase confidence in the cell counting results.
We have employed the guidance from ISO 20391 – 2:2019 Biotechnology – Cell Counting – Part 2 and utilized information from the ICH Q2 (R1) to develop an appropriate protocol to evaluate the performance of selected cell counting methods. The ICH Q2 (R1) guidance document presents multiple parameters for the validation of analytical methods, such as robustness, linearity, detection range, limits of detection (LOD), limits of quantitation (LOQ), precision (repeatability, intermediate precision, reproducibility), and accuracy. It is important to note that since there is no reference material to provide a reference value for cell concentration, the evaluation of the accuracy parameter needs to be indirectly assessed by orthogonal comparative methods. The ISO Cell Counting Standard Part 2 describes a detailed protocol to simultaneously evaluate precision (repeatability), coefficient of determination (R^{2}), and proportionality.
Utilizing the ISO Cell Counting Standard Part 2 document, we have identified several key parameters that can quickly and sufficiently assess the performance of cell counting methods [17]Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21., [18]Sarkar S, Pierce L, LinGibson S, Lund SP. Standards Landscape in Cell Counting: Implications for Cell & Gene Therapy. Cell Gene Ther. Ins. 2019; 5(1): 117–31.Sarkar S, Pierce L, LinGibson S, Lund SP. Standards Landscape in Cell Counting: Implications for Cell & Gene Therapy. Cell Gene Ther. Ins. 2019; 5(1): 117–31.. In this work, we will focus on an experimental protocol derived from the ISO Cell Counting Standard Part 2, which evaluates the coefficient of determination (R^{2}[CV)], [and proportionality index (]PI) of a cell counting method. The proportionality index is a metric introduced in the ISO Cell Counting Standard Part 2 that quantifies the degree to which a cell counting method conforms to the principle of proportionality, where it is expected that cell counts will scale proportionally with dilution. The principle of proportionality is a fundamental property of any cell counting method, and any deviation from proportionality would indicate a systematic or nonsystematic error resulting in a loss of measurement accuracy. To more directly evaluate systematic deviation from proportionality, which is an indicator of loss of accuracy, the PI is calculated by fitting a proportional model to the dilution series data, then summarizing residuals based on smoothed data, thus reducing the influence of random variation on the evaluation of proportionality [17]Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.. There are several approaches to calculate PI, where some PI metrics may be more relevant based on the fitforpurpose need of the cell counting method. Some metrics penalize more for outliers, while others weigh errors evenly across the dilutions or allow more contribution by higher cell concentrations. In this work, we utilized the PI model published previously from the National Institute of Standards and Technology (NIST) [17]Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.. It should be noted that sources of systematic error which are proportional to sample dilution will not be detected with this approach. For example, if debris are mixed with the cell suspension and falsely identified as cells, concentration of both cells and debris would be proportionally reduced with dilution, and the false counts would not affect the proportionality. In order to demonstrate the appropriate usage of the proposed experiments, these protocols were tested using various image cytometry systems from Nexcelom Bioscience LLC. (Lawrence, MA).
Comparative analysis methods can be employed to compare the performance of different cell counting methods. While the lack of reference material precludes the direct measurement of cell counting accuracy, comparison of orthogonal methods may serve as a viable alternative. It is also often desirable to determine how closely the results of one method will agree with another, such as when an instrument is upgraded after many years in the lab. One useful method is the construction of a Tukey meandifference plot, also known as a BlandAltman plot [20]Altman DG, Bland JM. Measurement in Medicine: The Analysis of Method Comparison Studies. J. Royal Stat. Soc. Series D (The Statistician) 1983; 32(3): 307–17., [21]Bland JM, Altman DG. Statistical Methods For Assessing Agreement Between Two Methods of Clinical Measurement. Lancet 1986; 327(8476): 307–10., [22]Tholudur A, Giron L, Alam K et al. Comparing Automated and Manual Cell Counts for Cell Culture Applications. BioProcess Int. 2006.. The BlandAltman analysis results in the calculation of a bias (with corresponding confidence interval) between two methods, indicating which method counts higher or lower on average and by how much. The analysis also provides an estimate of how well the two methods are expected to agree for a single sample. Here we modify the dilution series experimental design described in ISO Cell Counting Standard Part 2 document to collect data appropriate for a BlandAltman analysis while also meeting the standards requirements for calculating CV, R^{2}, and PI.
Usually, BlandAltman plots consist of absolute differences between two measurements plotted against their mean. In the case of cell counting, variance is not constant for different concentrations, but is generally proportional to the number of cells counted [17]Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21.. For BlandAltman analysis to be useful in such an application, the data can be transformed to achieve roughly constant variance across a range of concentrations. In this protocol, we use percent differences rather than absolute differences to achieve more uniform variance.
The BlandAltman analysis method produces three metrics of comparison:
The bias describes the average difference between measurement results obtained via the two methods. Due to biological variation in the samples and variability in the measurement process for both methods, it is impossible to predict exactly how much the measurement of any single sample will differ between the two methods. However, when measurements of many samples are averaged, a bias – even a slight one – may become clear. The bias may be interpreted as one method measuring higher or lower than another on average, though the difference between methods when measuring a single sample may vary widely.
The limits of agreement describe how widely these differences may vary. When added to and subtracted from the bias, the LoA define a range within which the difference between the measurements from two methods of a single sample is expected to be found. In this protocol, we use the limits of agreement that approximate a 95% confidence interval (1.96 × standard deviations) for a normal distribution. This is a sufficient approximation for our purposes and 72measurement sample size. If fewer measurements are acquired, confidence intervals calculated from the appropriate t distribution (rather than the Normal distribution) are advised. If the percent differences between the results from the two methods follow a normal distribution, we can expect that 95% of the differences will fall within one LoA from the value of the bias. In reality, the values will not be strictly normal, but the approximation is useful for evaluating subsequent measurements [23]Carkeet A. A Review of the Use of Confidence Intervals for BlandAltman Limits of Agreement in Optometry and Vision Science. Optometry Vision Sci. 2020; 97(1): 3–8.. If more statistical rigor is required, tests for normality can be applied, and the confidence intervals can be more exactly calculated [24]Carkeet A. Exact Parametric Confidence Intervals for BlandAltman Limits of Agreement. Optometry Vision Sci. 2015; 92(3): e71–e80.. Depending on the variation observed between samples relative to the variation between replicate measurements from each sample, it may be helpful to include random effects terms typically included in analysis of hierarchical experiments. In this work, we were not concerned with the sampletosample variation in the proposed experiments.
The confidence interval of the bias provides the approximate uncertainty for the calculated bias value and suggests a range within which the true value of the bias between the two cell counting methods is likely to be found. Unlike the LoA, this confidence interval narrows with an increased number of samples measured. If the 95% confidence interval is larger than the absolute value of the bias (i.e. the CI brackets the value 0), the method comparison has not demonstrated a statistically significant bias between the two methods (at α = 0.05 significance level). With enough samples, even a very slight bias may be confidently measured. A slight bias is often negligible compared to sample variation. Researchers should consider how a cell count is being used in order to determine acceptable levels of bias in their case.
Before proceeding with BlandAltman comparative analysis for cell counting, researchers should:
A higher number of paired measurements can reduce the uncertainties of the bias and LoA, e.g. the confidence interval of the bias narrows with more measurements. Researchers should determine the precision they require, and increase the number of paired measurements accordingly – we suggest a minimum of 20 paired measurements be used as a starting point. Finally, it is possible that either the bias or the variation will vary with cell concentration. In such a case, the bias and LoA obtained for the entire group of data may not be representative of how the two methods compare over a narrower range of concentrations. It may be useful to perform BlandAltman analysis on smaller subsets of data.
The cell counting method performance evaluation and comparison protocols can be applied to research, analytical method development, process development, and preclinical or clinical trials. In addition, the method can be applied to a plethora of research fields requiring the usage of cells such as cellular and gene therapy, immunooncology and immunotherapy, cell line development and biologics production, virology and infectious disease, regenerative medicine, toxicology, food science, and even renewable energy. The quality of cell counting results is critical for a wide range of cell types used in the research fields mentioned above. These cell types can include primary cells such as human or mouse whole blood, cord blood, bone marrow aspirate, adipose tissue, hepatocytes, PBMCs, leukapheresis sample, platelets, tumor or tissue digests are typically used. In addition, bacteria and yeast cells are often used to generate biologics or used for beverage production.
The cell counting method performance evaluation proposed here consists of a dilution series experiment and comparative analysis for multiple methods. The experimental design is demonstrated using CHOS and Jurkat cell lines fluorescently stained with acridine orange and a green nuclear dye. Two cell counting systems are compared: the Cellaca MX HighThroughput Cell Counter (Cellaca MX) and the Celigo Image Cytometer (Celigo). It is important to note that ISO Cell Counting Part 2 requires users to assess pipetting error contributions to dilution integrity to establish confidence in dilution and sampling. Here, we conducted a preevaluation of pipetting error, which will not be described in this protocol. It is also important to investigate the stability of the target cell sample prior to conducting the experiment in order to avoid drift in concentration and viability during the assay time frame. The stability of the Jurkat and CHO cells used in this work have been previously tested and showed no noticeable trends (Supplementary Figure 1).
The dilution experiment consists of a 6point concentration series of the target cell types, where each concentration is independently produced from the original stock (rather than the other dilutions) to reduce a propagation of dilution error that can affect proportionality. The dilution series should span the typical concentration range of the target cell samples in order to evaluate the performance of the cell counting method in the specified range.
Three replicate samples are generated per concentration, and each replicate sample is measured 4 times per cell counting method so that each method provides a total of 12 measurements per concentration and a total of 72 measurements in a 6point concentration series. The measurements are used to calculate the coefficient of determination (R^{2}), precision (repeatability – Coefficient of Variation, CV), and proportionality index (PI) parameters for each cell counting method. It is important to note that the tested Jurkat and CHO cells were stained with acridine orange and Nuclear Green dye to measure only the total cell concentration in this work.
For performance comparison between two cell counting methods, the BlandAltman method is applied. Like the proportionality measurement, the comparison results are valid only for the intended use of the specific methods (cell type, assay type, exact instruments, etc.), and only for the range of cell concentrations included in the test. Therefore, it is vital to first define the exact methods, test conditions, and range of cell concentrations over which comparison is desired. For most accurate results, the BlandAltman analysis should include as many measurements as possible, encompassing the sources of variation that are expected for the normal operation of the cell counting method, such as multiple operators, reagent lots, and cell culture flasks. Each point on the BlandAltman plot is obtained by using both cell counting methods to measure a single sample. The sample should be carefully mixed to ensure homogeneity before portions are taken for measurement with each cell counting method. Measurements should be made with minimal lag time between them, simultaneously if possible. If the experiment described above is performed with the same tubes of cells using both methods at the same time, BlandAltman analysis may be performed with the resulting data. If desired, tighter confidence intervals on the calculated bias or less uncertainty on the Limits of Agreement can be obtained by supplementing the data with more samples. Concentrations spanning the selected concentration range should be represented roughly equally in the samples used.
In general, the expertise required to implement the cell counting method performance evaluation is proper training by an expert user in the operation of the cell counting systems. In addition, the users should be trained on sample preparation to ensure consistent performance of the dilution, sampling, and staining steps of the cell counting process.
Accuracy is one of the most critical parameters for the validation of an analytical method, however, it cannot be directly applied to most cell counting methods. Since there are limited live cell reference standards, it is challenging to assess the accuracy of a cell counting method. Therefore, proportionality is an alternative parameter to assess accuracy relative to dilution fraction, which serves as the internal control, as well as utilizing orthogonal methods for comparison.
It should be recognized that R^{2} values calculated over a range of concentrations are strongly dependent on the range chosen. A larger range of linear data results in an R^{2} value closer to 1. If comparison between R^{2} values is to be attempted, it is important that the range for the two calculations be the same. In addition, it should be noted that the proportionality index as defined here is not normalized to the number of dilution fractions and the number of biological replicates per dilution fraction. It is required that the same experimental design be used if PI is to be meaningfully compared between methods.
Prepare the PBS solution by mixing 5 L of H2O with 1 packet of PBS powder to generate a solution of 0.01M PBS at pH 7.4 with NaCl at 0.138 M and KCl at 0.0027 M.
Prepare CHOS medium (500 mL) with the CD CHO Medium (1X) and supplement with 5 mL of the GlutaMAX1 (100X) and 5 mL of the HT Supplement (100X).
Prepare Jurkat medium (500 mL) with the RPMI Medium 1640 (1X) and supplement with 10% FBS (50 mL) and 5 mL of the Antibiotic Antimycotic Solution (100X).
The acridine orange (AO) and propidium iodide (PI) staining solution is already prepared to the correct concentration before staining at 1:1 with the cells.
The acridine orange (AO) staining solution is already prepared to the correct concentration before staining 1:1 with the cells.
Prepare a 2X staining solution (10 µM) by mixing PBS and the Nuclear Green stock solution at 5 mM. Pipette 10 mL of PBS into a 15mL centrifuge tube and add 20 µL of the Total Cell Nuclear Green stock solution. Close the 15mL centrifuge tube and invert 10X to mix the staining solution before use.
Connect the Cellometer Spectrum to the operating laptop computer via the USB cable and plug in the power cord. Turn the instrument power on from the back side and then open the Cellometer Spectrum analysis software. In the Cellometer Spectrum software, select the “AOPI Viability Assay_S5” default assay type for cell counting.
Connect the Cellaca MX to the operating laptop computer via the USB cable and plug in the power cord. Turn the instrument power on from the back side and then open the Cellaca MX analysis software. In the Cellaca MX software (v1.2), select the “MX04.0_AOPI_LiveDead” default assay type for cell counting.
Turn on the Celigo power on the front and open the Celigo analysis software. Navigate to the top right and click on ‘Administration’ and then select ‘Manage Plate Profiles’. After the “Plate Profile Management” window opens, click on the ‘Import’ button and select the plate profile for Cellaca 12 × 2 plate. Return to the home screen for image acquisition and analysis.
Timing: 20 – 30 min for passaging the cells and measuring their concentration and viability.
Timing: 20–30 min for passaging the cells and measuring their concentration and viability.
Timing: 15 min for collecting the cells from cell culture flasks, 5 min for cell counting and viability analysis, and 10 min for adjusting cell sample concentration if necessary.
Timing: 15–30 min with a single, manual pipette for sample preparation. 15–20 min for incubation of cell samples mixed with Nuclear Green (Figure 1Cell sample preparation and measurement process diagram.Procedure sequence from left to right: (1) Collect your target cell sample and prepare different concentrations with specific dilution fractions using cell media. (2) Repeat this process to generate 3 replicates for each dilution fraction. (3) Label each tube in random order from 1–18. (4) Prepare and measure each tube 4 times with each selected cell counting method. (5) Analyze the images with each cell counting method to generate cell counting results. (6) Utilize the cell counting results to generate proportionality index (PI), coefficient of variation (CV), and coefficient of determination (R^{2}).). ∼10 min per Cellaca MX plate with a single, manual pipette for cell counting preparation.
CRITICAL: Under the guidance of ISO Cell Counting Part 2, an initial accuracy validation experiment of pipetting volume using the experimental pipettors is necessary to increase sampling confidence. Such validation can be performed with a sensitive and wellcalibrated laboratory balance and a fluid of known density, but the procedure will not be described in this protocol. Directly dilute the cells to generate independent dilution samples instead of serial dilution to reduce the propagation of pipetting error that can affect proportionality.
Table 1 Dilution fractions and the corresponding volumes preparation for cell sample and PBS.  
DF  Cell volume (µL)  PBS volume (µL) 
1.0  120  0 
0.9  108  12 
0.7  84  36 
0.5  60  60 
0.3  36  84 
0.1  12  108 
Table 2 Cellaca plate map for cell samples at different DFs.  
Plate 1  
1  2  3  4  5  6  7  8  9  10  11  12  
A  0.1  0.1  0.1  0.1  0.3  0.3  0.3  0.3  0.5  0.5  0.5  0.5 
B  0.7  0.7  0.7  0.7  0.9  0.9  0.9  0.9  1.0  1.0  1.0  1.0 
Plate 2  
1  2  3  4  5  6  7  8  9  10  11  12  
A  0.1  0.1  0.1  0.1  0.3  0.3  0.3  0.3  0.5  0.5  0.5  0.5 
B  0.7  0.7  0.7  0.7  0.9  0.9  0.9  0.9  1.0  1.0  1.0  1.0 
Plate 3  
1  2  3  4  5  6  7  8  9  10  11  12  
A  0.1  0.1  0.1  0.1  0.3  0.3  0.3  0.3  0.5  0.5  0.5  0.5 
B  0.7  0.7  0.7  0.7  0.9  0.9  0.9  0.9  1.0  1.0  1.0  1.0 
Timing: Scanning and analysis are 6 min per plate for the Cellaca MX and 5–10 min per plate for the Celigo.
Timing: ∼30 min to calculate and analyze the parameters for performance evaluation for one cell line and one stain.
[1]
, where M_{Air} is the concentration acquired with method A for sample i during replicate measurement r.
[2]
[3]
[4]
[5]
[6]
[7]
[8]
Timing: ∼30 min to analyze and plot the BlandAltman comparison data for one cell line and one stain.
[9]
, where X_{i} is the sample mean given by
[10]
, where X_{ir} is the sample mean given by
Calculate the bias from method A to method B (Bias_{AB}) by averaging the Y_{i} values using Equation 11 or by averaging the Y_{ir} values using Equation 12
[11]
, where N is the number of samples (for paired samples, unpaired replicates, i.e. for each sample, different replicates are measured with each method).
[12]
, where N is the total number of replicate measurements (for paired samples with paired replicates, i.e. for each sample, the same replicates are measured using both methods).
[13]
where N is the number of samples (paired samples, unpaired replicates).
[14]
where N is the total number of replicate measurements (paired samples, unpaired replicates).
[15]
, where N is the number of samples (for paired samples without paired replicates) or the total number of replicate measurements (if both samples and replicates are paired).
BlandAltman comparative method: graphical representation
Follow the troubleshooting Table 3 to optimize the experiments and output.
Table 3 Troubleshooting table.  
Step  Problem  Possible reason  Solution 
62, 63  CV is too large at one or a few DFs  Sampling or pipetting error 

Counting errors due to clumps 
 
67  Poor Proportionality  Propagation of pipetting error 

68, 70–78  A large bias between two cell counting methods  Sample variation (i.e. different stocks of samples) 

Sample condition change (e.g. photobleaching, sample dryout) 
 
Cell counting analysis variation (e.g. declumping) 
 
Instrument comparison 
 
Instrument calibration 

Two cell lines (CHOS, Jurkat), two dyes (AO, Nuclear Green), and 2 cell counting methods were evaluated to demonstrate the application of cell counting method performance evaluation and BlandAltman comparative analysis. Figure 2Cell counting results for (a) mean concentrations and (b) pooled CV of 6point concentration series of Jurkat Cells Stained with AO.The mean concentrations and CV values measured by Cellaca MX and Celigo were highly comparable at each dilution fraction. It is clear that as concentration decreased, the pooled CV increased, likely due to the Poisson Noise (Random Error) at lower concentrations. shows the mean and pooled CV, respectively, of the 6point concentration series of Jurkat cells stained with AO using both cell counting systems. Table 4 shows the numerical results for the mean and pooled CV. The concentration range measured in the experiment was ~5 × 10^{5} to ~6 × 10^{6} cells/mL. Both Cellaca MX and Celigo have pooled CVs ranging from 1.8–7.6% for all replicates per concentration.
Table 4 Calculated mean concentration and pooled CV for each dilution fraction using ISO Cell Counting Standard Part 2.  
Cell counting method  DF  n  Mean (cells/mL)  Pooled CV (%) 
Cellaca MX  0.1  12  5.37E+05  7.5 
0.3  12  1.78E+06  4.2  
0.5  12  2.88E+06  3.3  
0.7  12  3.93E+06  2.2  
0.9  12  5.05E+06  3.8  
1.0  12  5.74E+06  2.3  
Celigo  0.1  12  5.21E+05  7.6 
0.3  12  1.71E+06  3.9  
0.5  12  2.80E+06  3.6  
0.7  12  3.91E+06  2.7  
0.9  12  5.06E+06  3.6  
1.0  12  5.80E+06  1.8 
Based on the measurements in Table 4, the coefficient of determination and proportionality index can be calculated via regression analysis. Figure 3Linear regression fitting of the 6point concentration series as a function of dilution fractions (DFs), which shows distribution of concentration measurements for Cellaca MX and Celigo at each dilution fraction. shows the proportional fits of the 6point concentration series as a function of dilution fractions for both Cellaca MX and Celigo. Results for each parameter are shown in Figure 4Comparison of the R^{2} and PI values between Cellaca MX and Celigo. The R^{2} and PI values are both highly comparable. and Table 5. Both Cellaca MX and Celigo show comparable values of coefficient of determination (R^{2} values) and proportionallity indices (PIs) from the proportional fits in this cell counting method evaluation. No significant differences are observed between Cellaca MX and Celigo for both R^{2} and PI values.
Table 5 Calculated Rsquare and PI values for performance evaluation.  
RSquare  PI  
Cellaca MX  0.997  0.44 
Celigo  0.997  0.42 
Significance  No  No 
Next, the BlandAltman method is applied to compare the performance between Cellaca MX and Celigo cell counting methods. Figure 5BlandAltman plot between Cellaca MX and Celigo cell counting methods.The calculated percent differences show an increasing trend as the concentration increases. shows a representative BlandAltman plot between Cellaca MX and Celigo. In this plot, a positive percentage indicates a higher concentration for Celigo measurements. Each point in the plot represents a pair of measurements determined by both cell counting methods. Results of concentration bias, 95% confidence interval and standard deviation of the bias between Cellaca MX and Celigo are shown in Table 6. A bias of ~ 1.5% (n = 72 replicate measurements) indicates that concentration measured by Celigo is
.5% lower than Cellaca MX in this cell counting method comparison. Since the value of 0 lies outside the confidence interval of the bias, it is concluded that the concentration difference observed between these two cell counting methods is significant (p < 0.05), despite being relatively small. The bias exhibits a slight dependence on cell concentration in this case. An additional paired measurement made using two methods under the same conditions would be expected to fall within the range defined by the LoA’s in approximately 95% of cases.
Table 6 BlandAltman comparative analysis results between Cellaca MX and Celigo.  
Bias  Limit of agreement  CI of bias 
1.5%  6.9% to 3.9%  2.1% to 0.8% 
A summary of the cell counting performance evaluation and comparison results between Cellaca MX and Celigo using different combinations of targeted cell lines and staining solutions is shown in Table 7. Concentration biases are 2–6% for the paired cell counting methods in these four cell counting methods. Because each cellstain combination was treated independently, the results were not combined, and the False Discovery Rate (FDR) or FamilyWise Error Rate (FWER) were not calculated. Overall, we have conducted multiple practical experiments following ISO Cell Counting Standard Part 2 with Celigo and Cellaca MX, and the results presented here are in the expected range of the cell counting measurement quality.
Table 7 Summary table of cell counting performance evaluation and comparison results.  
Cell line  Staining solution  Cellaca MX  Celigo  Bias  LoA  95% CI of bias  Significance of bias  
R^{2}  Pooled CV range (%)  PI  R^{2}  Pooled CV range (%)  PI  
Jurkat  Nuclear green  0.998  3.8% to 6.1%  0.30  0.997  3.7% to 7.4%  0.37  3.4%  6.0% to 12.8%  2.3% to 4.5%  Y 
Jurkat  AO  0.997  2.2% to 7.5%  0.44  0.997  1.8% to 7.6%  0.42  1.5%  6.9% to 3.9%  2.1% to 0.8%  Y 
CHO  Nuclear green  0.998  3.4% to 5.8%  0.40  0.999  2.7% to 6.8%  0.35  5.6%  3.4% to 14.6%  4.5% to 6.7%  Y 
CHO  AO  0.998  2.7% to 7.0%  0.44  0.996  2.7% to 6.4%  0.35  5.1%  2.4% to 12.5%  4.2% to 5.9%  Y 
The ISO Cell Counting Standard Part 2 enables cell and gene therapy researchers to conduct experiments to evaluate and compare the quality of cell counting measurement processes. In this practical application of the standard, we demonstrated the evaluation of the Cellaca MX and Celigo using Jurkat and CHO cells tailored to the bioprocessing and cell therapy communities. The users of the ISO Cell Counting Standard Part 2 may utilize the protocol to evaluate one or more cell counting methods. For the evaluation of one method, one can perform the experiments to establish a baseline for each of the quality parameters, where this baseline can be further monitored with different operators, instruments, processes, etc. For comparison of cell counting methods, the quality parameters can be compared via bootstrap analysis or replicate studies, and the difference or bias between the methods can be determined using the BlandAltman comparative analysis. It is also important to note that the quality parameters obtained from the ISO Cell Counting Standard Part 2 are specific to the measurement process evaluated (e.g. operators, instruments, cell sample properties, etc.), thus the robustness of the measurement process should also be evaluated in order to extend the findings of the study to similar measurement processes.
1. Boyiadzis MM, Dhodapkar MV, Brentjens RJ et al. Chimeric antigen receptor (CAR) T therapies for the treatment of hematologic malignancies: clinical perspective and significance. J. ImmunoTher. Cancer 2018; 6(137): 1–12. Crossref
2. Gonçalves GAR, Paiva RdMA. Gene therapy: advances, challenges and perspectives. Einstein 2017; 15(3): 369–75. Crossref
3. Huang R, Li X, He Y et al. Recent advances in CART cell engineering. J. Hematol. Oncol. 2020; 13(86): 1–19. Crossref
4. Li Y, Huo Y, Yu L, Wang J. Quality Control and Nonclinical Research on CART Cell Products: General Principles and Key Issues. Engineering 2019; 5: 122–31. Crossref
5. Seimetz D, Heller K, Richter J. Approval of First CARTs: Have we Solved all Hurdles for ATMPs? Cell Med. 2019; 11: 1–16. Crossref
6. Nagata S, Hanayama R, Kawane K. Autoimmunity and the Clearance of Dead Cells. Cell 2010; 140(5): 619–30. Crossref
7. Rock KL, Kono H. The inflammatory response to cell death. Ann. Rev. Pathol. 2008; 3: 99–126. Crossref
8. Zhang Y, Chen X, Gueydan C, Han J. Plasma membrane changes during programmed cell deaths. Cell Res. 2018; 28: 9–21. Crossref
9. 21st Century Cures. United States of America; 2016.
10. Arcidiacono JA, Bauer SR, Kaplan DS, Allocca CM, Sarkar S, LinGibson S. FDA and NIST collaboration on standards development activities supporting innovation and translation of regenerative medicine products. Cytotherapy 2018; 20: 779–84. Crossref
11. LinGibson S, Sarkar S, Elliott JT. Summary of the National Institute of Standards and Technology and US Food And Drug Administration cell counting workshop: Sharing practices in cell counting measurements. Cytotherapy 2018; 20(6): 785–95. Crossref
12. Biotechnology – Cell counting – Part 1: General guidance on cell counting methods. In: Standardization IOf, editor; 2018. Crossref
13. Biotechnology – Cell counting – Part 2: Experimental design and statistical analysis to quantify counting method performance. In: Standardization IOf, editor; 2019. Crossref
14. Validation of Analytical Procedures: Text and Methodology Q2(R1). In: Guideline IHT, editor; 2005. Crossref
15. Guidance for Industry: Process Validation: General Principles and Practices. In: Services USDoHaH, Administration FaD, (CDER) CfDEaR, (CBER) CfBEaR, (CVM) CfVM, editors; 2011. Crossref
16. Guidance for Industry: Analytical Procedures and Methods Validation for Drugs and Biologics. In: Services USDoHaH, Administration FaD, (CDER) CfDEaR, (CBER) CfBEaR, editors; 2015. Crossref
17. Sarkar S, Lund SP, Vyzasatya R, Vanguri P, Elliott JT, Plant AL, LinGibson S. Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Cytotherapy 2017; 19(12): 1509–21. Crossref
18. Sarkar S, Pierce L, LinGibson S, Lund SP. Standards Landscape in Cell Counting: Implications for Cell & Gene Therapy. Cell Gene Ther. Ins. 2019; 5(1): 117–31. Crossref
19. Cohen J. The earth is round (p<.05). Am. Psychol. 1994; 49(12): 997–1003.
20. Altman DG, Bland JM. Measurement in Medicine: The Analysis of Method Comparison Studies. J. Royal Stat. Soc. Series D (The Statistician) 1983; 32(3): 307–17. Crossref
21. Bland JM, Altman DG. Statistical Methods For Assessing Agreement Between Two Methods of Clinical Measurement. Lancet 1986; 327(8476): 307–10. Crossref
22. Tholudur A, Giron L, Alam K et al. Comparing Automated and Manual Cell Counts for Cell Culture Applications. BioProcess Int. 2006. Crossref
23. Carkeet A. A Review of the Use of Confidence Intervals for BlandAltman Limits of Agreement in Optometry and Vision Science. Optometry Vision Sci. 2020; 97(1): 3–8. Crossref
24. Carkeet A. Exact Parametric Confidence Intervals for BlandAltman Limits of Agreement. Optometry Vision Sci. 2015; 92(3): e71–e80. Crossref
25. AbuArafeh A, Jordan H, Drummond G. Reporting of method comparison studies: a review of advice, an assessment of current practice, and specific suggestions for future reports. Br. J. Anaesthesia 2016; 117(5): 569–75. Crossref
26. Navidi W. Statistics for Engineers and Scientists. New York: McGrawHill; 2013. Crossref
27. Sokal RR, Rohlf FJ. Biometry. New York: W. H. Freeman and Company; 1969. Crossref
Yongyang Huang
Department of Advanced Technology R&D, Nexcelom Bioscience LLC, Lawrence, MA, USA
Jordan Bell
Department of Advanced Technology R&D, Nexcelom Bioscience LLC, Lawrence, MA, USA
Dmitry Kuksin
Department of Advanced Technology R&D, Nexcelom Bioscience LLC, Lawrence, MA, USA
Sumona Sarkar
Biosystems and Biomaterials Division, Materials Measurement Laboratory, National Institute of Standards and Technology, Gaithersburg, MD, USA
Laura T Pierce
Biosystems and Biomaterials Division, Materials Measurement Laboratory, National Institute of Standards and Technology, Gaithersburg, MD, USA
David Newton
Statistical Engineering Division, Information Technology Laboratory, National Institute of Standards and Technology, Boulder, CO, USA
Jean Qiu
Department of Advanced Technology R&D, Nexcelom Bioscience LLC, Lawrence, MA, USA
Leo LiYing Chan
Author for correspondence:
Department of Advanced Technology R&D, Nexcelom Bioscience LLC, Lawrence, MA, USA
Contributions: Leo LiYing Chan and Jean Qiu conceived the study and experimental design. Dmitry Kuksin performed the cell culture and provided the Jurkat and CHOS cells for the experimentation. Jordan Bell and Yongyang Huang developed software application for the ISO Cell Counting Standards Part II parameters and the BlandAltman analysis method. Yongyang Huang conducted the cell counting method performance evaluation and comparison experiments. Sumona Sarkar and Laura T Pierce contributed to the development of cell counting standardization, which this protocol was derived from. David Newton contributed to the development and description of the statistical analysis. Leo LiYing Chan, Yongyang Huang, Jordan Bell, Sumona Sarkar, and Laura T Pierce wrote the manuscript.
Acknowledgements: The authors would like to thank Pauline N. Mitchell for providing the art work of the Cell sample preparation and measurement process diagram.
Disclosure and potential conflicts of interest: Yongyang Huang, Jordan Bell, Dmitry Kuksin, Jean Qiu, and Leo LiYing Chan declare competing interests. The application of the cell counting method performance evaluation and comparison employed cell counting systems and reagents from Nexcelom Bioscience LLC.
NIST Copyright disclaimer: Official contribution of the National Institute of Standards and Technology; not subject to copyright in the United States.
NIST Commercial product disclaimer: Certain commercial equipment, instruments, or materials are identified in this paper in order to specify the experimental procedure adequately. Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended to imply that the materials or equipment identified are necessarily the best available for the purpose.
Funding declaration: The authors received no financial support for the research, authorship and/or publication of this article.
Copyright: Published by Cell and Gene Therapy Insights under Creative Commons License Deed CC BY NC ND 4.0 which allows anyone to copy, distribute, and transmit the article provided it is properly attributed in the manner specified below. No commercial use without permission.
Attribution: Copyright © 2021 Nexcelom Bioscience. Published by Cell and Gene Therapy Insights under Creative Commons License Deed CC BY NC ND 4.0.
Article source: Invited; externally peer reviewed.
Submitted for peer review: Jul 2 2021; Revised manuscript received: Aug 19 2021; Publication date: Sep 15 2021.