"QuizID"	"AppID"	"AppTitle"	"TaskID"	"TaskText"	"RecordID"	"Record"	"Type"	"Note"
"DSAIRM_fitbasicmodel"	"fitbasicmodel"	"Basic Virus Model Fitting"	1	"Start with 1e6 uninfected cells, no infected cells, 1 virion (assumed to be in the same units of the data, TCID50/ml), no uninfected cell birth and deaths, lifespan of infected cells 12 hours (make sure to convert to a rate), and unit conversion factor to 0. Set virus production rate to 0.001 with lower and upper bounds of 0.0001 and 100; _psim_ can be anything for now. Set the infection rate to _b_ = 0.1 with lower/upper bounds of 0.001/10; _bsim_ can be anything. Set the virus decay rate to 1 with lower/upper bounds of 0.01/100; _dVsim_ can be anything.

The parameters *p*, *b* and *d~V~* are being fit, the values we specify here are the starting conditions for the optimizer and the upper and lower bounds which the parameters must remain inside. Note that if the lower bound is not lower than / equal to and the upper not higher than / equal to the parameter, you will get an error message when you try to run the model. We can ignore the values for simulated data for now; to do so, set _usesimdata_ to 0. This also means the _noise_ variable is ignored, i.e., you can set it to anything. 


Start with a maximum of 1 iteration/fitting step for the optimizer and _solvertype_ = 1. Choose to plot the y-axis on a log scale using either ggplot or plotly. Run the simulation. 


Since you only do a single iteration, nothing is really optimized. We are just doing this so you can see the time-series produced with the starting conditions we picked for the parameters. Notice that the virus load predicted by the model and the data are already fairly close. Look at the SSR value reported underneath the plot. It should be 3.25. As we fit, if things work ok, this value will go down, indicating an improved fit.


Now choose _iter_ = 20, i.e., fit for 20 iterations. Run the simulation. Look at the results. The plot shows the final fit. The model-predicted virus curve will be closer to the data. Record the SSR value; it should have gone down indicating a better fit. Also, printed below the figure are the values of the fitted parameters at the end of the fitting process. They should differ from the values you started with."	"T1R1"	"SSR after 20 interations (round to 2 digits)."	"Rounded_Numeric"	"Round to two significant digits"
"DSAIRM_fitbasicmodel"	"fitbasicmodel"	"Basic Virus Model Fitting"	2	"Repeat the same process, now fitting for 40 iterations. You should see some more improvement in the SSR, meaning a further improvement in fit. A fitting step or iteration is essentially a try of the underlying code to find the best possible model. Increasing the tries/iterations usually improves the fit until the solver reached a point where it can't improve further. This is the best fit, though see the next tasks for a big caveat. In practice, one should not specify a fixed number of iterations; that is just done here so things run reasonably fast. Instead, one should ask the solver to run as long as it takes until it can't find a way to further improve the fit (in our case can't further reduce the SSR). The technical expression for this is that the solver has converged to the solution. In this example, it happens quickly. Keep increasing iterations until you find no further reduction in SSR."	"T2R1"	"SSR after 40 interations (round to 2 digits)."	"Rounded_Numeric"	"Round to two significant digits"
"DSAIRM_fitbasicmodel"	"fitbasicmodel"	"Basic Virus Model Fitting"	3	"The goal is to find the best fit, the one with the lowest SSR posssible (for a given model and data set). While in theory, there is always one (or maybe more than one) best fit, for the kind of models we are fitting here, it is often hard to numerically find this best fit (that's a big difference compared to many standard statistical models where the numeric routine is fairly straightforward). 


To explore this, start with the same input values as for task 1, run the optimizer for 40 steps but now use solver/optimizer type 2. Then, repeat for type 3. You should find that solver 2 is at an SSR after 40 steps that's above solver 1 while solver 3 found a value that's below what type 1 is able to find."	"T3R1"	"SSR after 40 interations, solver type 3 (round to 2 digits)."	"Rounded_Numeric"	"Round to two significant digits"
"DSAIRM_fitbasicmodel"	"fitbasicmodel"	"Basic Virus Model Fitting"	4	"We just saw that after a few iterations, different numerical solvers/optimizers obtain different results. That's maybe not surprising and we would not care much about that (apart from trying to find the fastest one), as long as they can all find the same best fit given enough time/iterations. Unfortunately, this doesn't always happen. Let's, explore this. Run each solver for 1000 iterations, then 2000. Depending on the speed of your computer, this might take a while, so be patient. You should find that each solver produces the same SSR for 1000 and 2000 steps, so each solver has converged (reached what it thinks is the overall best fit). Unfortunately, the solvers don't agree on what the best fit is. In this example, solver 2 performs best, with an SSR slightly below 1. This illustrates the challenge when fitting the kind of models we have here, namely that we often have to explore different solvers (and starting conditions, we'll get there) to be reasonably sure we can find the overall best fit. This general problem gets worse the more parameters you try to fit."	"T4R1"	"SSR after 1000 interations, solver type 2 (round to 2 digits)."	"Rounded_Numeric"	"Round to two significant digits"
"DSAIRM_fitbasicmodel"	"fitbasicmodel"	"Basic Virus Model Fitting"	5	"To explore the role that different starting values for the estimated parameters can have, set _p_=0.01, _b_=0.01, and _d~V~_=10. Run a single iteration with solver 1. SSR should be 3.54, similar but not the same as for the start of task 1. 


Now, run each of the 3 solvers for 1000 iterations. You'll find that all of them are able to find better fits than previously. Note that the scientific question has not changed: the model and the data are the same and we are still trying to find the combination of parameter values that give the best agreement between model and data. All we have changed is giving the optimizers a different starting position from which to go and find the best fit. In many basic statistical models, you can start anywhere and always arrive at the same answer. This is not the case here. For the kind of fitting problem we usually encounter with our simulation models, optimizers might not find the overall best fit, even if you run them until they converge. What can happen is that a solver converges to a local optimum, which is a combination of parameters that produces a good result, and, as the solver tries to change the parameter values, each result is worse; so, it determines it found the best fit. Unfortunately, there might exist a completely different combination of parameters that give an even better fit, but the solver can't find that solution. It is stuck in a local optimum. Many solvers, even so-called 'global' solvers, can get stuck. Unfortunately, we never know for certain if we found the best fit, and there is no one solver type that is best for all problems. So, in practice, what one needs to do is try different starting values and different solver/optimizer routines, and if many of them find a best fit with the lowest SSR, it's quite likely (though not guaranteed) that we found the overall best fit (lowest SSR)."	"T5R1"	"SSR after 1000 interations, solver type 2 (round to 2 digits)."	"Rounded_Numeric"	"Round to two significant digits"
"DSAIRM_fitbasicmodel"	"fitbasicmodel"	"Basic Virus Model Fitting"	6	"One major consideration when fitting these kind of mechanistic models to data is the balance between data availability and model complexity. The more and richer data one has (e.g., measurements for multiple model variables) the more parameters one can relibably estimate and therefore the more detailed a model can be. If one tries to ask too much from the data, it leads to the problem of overfitting, i.e., trying to estimate more parameters than can be robustly estimated for a given dataset (this is also known as the identifiability problem). This can show itself in the problems we've seen above, the solvers getting stuck in various local optimums, or finding more or less the same SSR for large ranges of parameters. While there are mathematical ways to test if a model can be estimated given the data, this only works for simple problems and doesn't consider the fact that most data are noisy. The most general, and usually best approach to try and safeguard against overfitting is to test if our model can in principle recover estimates in a scenario where parameter values are known. To do so, we can use our model with specific parameter values and simulate data. We can then fit the model to these simulated data. If everything works, we expect that, ideally independent of the starting values for our solver, we end up with estimated best-fit parameter values that agree with the ones we used to simulate the artificial data. We'll try this now with the app.

Set everything as in task 1 (i.e., hit the _Reset Inputs_ button). Then set _psim_ = 0.001, _bsim_ = 0.1, and _dVsim_ = 1, i.e., to the same values as the values used for starting the fitting routine (_p_/_b_/_dV_). Set iteration to 1, solver type 1, y-axis on log scale. Run the simulation. It should be the same as in task 1. Take a close look at the data in the plot. Then set _usesimdata_ = 1, run again for 1 fitting step. You should now see that the data has changed. Instead of the real data, we now use simulated data. Since the parameter values for the simulated data and the starting values for the fitting routine are the same, the time-series is on top of the data and the SSR is (up to rounding errors) 0."	"T6R1"	"TRUE or FALSE: If you set the starting values for your model fitting routine to those used to create the data, you'll get a perfect fit."	"Logical"	"Report TRUE or FALSE"
"DSAIRM_fitbasicmodel"	"fitbasicmodel"	"Basic Virus Model Fitting"	7	"Now we can test if we have a problem with overfitting (trying to estimate too many parameters given the data) by choosing starting values for our fitting routine that are different from those we used to produce the simulated data. We hope that after enough iterations, the solver will reach a best fit with SSR close to 0 and estimates for the parameters that agree with the ones we used to produce the data. 


Set _p_ = 0.01, _b_ = 0.01, _d~V~_ = 2. Don't change the _psim_/_bsim_/_dVsim_. Run simulation for 1 iteration. You'll see, as expected, a mismatch between model and simulated data with an SSR = 2.2. 


Now, run each solver for 500 iterations. You should find that solver 2 finds a best fit with SSR approximately 0 and the estimated parameters are those we used to generate the data. Solvers 1 and 3 can't quite find it. (You can try more iterations, but it seems those solvers are stuck. They might find the best fit with different starting conditions). This means that in this case, we seem to be able to estimate all parameters (as long as we play around with different solvers and starting conditions). You can play around with the values for the data simulation parameters and different starting values for the fitting routine. You'll likely find that sometimes your solver can accurately determine the parameter values, sometimes not. If it works for at least some starting values and some solvers, it means you should be able to estimate your model parameters. Of course with the caveats described above, i.e., it might require a good bit of testing/searching.


Since real data are always noisy, there is an option to add noise to your simulated data by choosing a non-zero value for _noise_. Explore a bit what happens if you do that. The more noise you add, the more disagreement will you get between best fit estimates for the parameter values and what you used to simulate the data. This is to be expected, since your data is now not coming straight from the simulation, but instead gets modified by noise."	"T7R1"	"SSR after 500 interations, solver type 1 (round to 2 digits)."	"Rounded_Numeric"	"Round to two significant digits"
"DSAIRM_fitbasicmodel"	"fitbasicmodel"	"Basic Virus Model Fitting"	8	"So far, we haven't talked much about the parameter bounds. In theory, you shouldn't need any bounds. If your model is a good one, you just start with some parameter values and your optimizer will find the best fit. As you already saw, in practice it's not that easy. If you don't give the optimizer any guidance as to what parameter values are reasonable, it might try all kinds of unrealistic values. This can slow down the fitting, it can make it more likely you get stuck in a local optimum, or your underlying simulation function fails because the optimizer is asking it to run the model for nonsensical values. All of these problems suggest that giving bounds for parameters is a good idea. Let's explore the impact of bounds briefly. 


Reset all inputs (that means we are switching back to the real data, i.e. _usesimdata_ = 0). Then set _p = b = d~V~ = 0.1_ as starting values. Run a single iteration to see that with these starting values, the model is far from the data (SSR = 91). Then, run 1000 interations of solver 2, you should find an okay fit with an SSR = 4.12. 


Now, change lower bounds to 1e-6 and upper bounds to 1e6 for all 3 parameters. Re-run the fitting. You'll find a much poorer fit with unreasonable parameter estimates. This is an example of how bounds help find the best fit. You can explore the impact of bounds further by changing their values, the starting values, the number of iterations and solver type. You'll find that sometimes wide bounds are okay and don't impact the fitting much, but, other times, things don't work with wide bounds.


One problem can arise when the best-fit value reported from the optimizer is the same as the lower or upper bound for that parameter. This likely means if you widen the bounds the fit will get better. However, the parameters have biological meanings and certain values do not make sense. For instance a lower bound for the virus decay rate of 0.001/day would mean an average virus lifespan of 1000 days or around 3 years, which is not reasonable for flu *in vivo*. That means if your best fit happens at the bounds, or in general for parameter values that make no biological sense, then your underlying model needs to be modified."	"T8R1"	"SSR after 1000 interations, solver 2, wide bounds (round to 2 digits)."	"Rounded_Numeric"	"Round to two significant digits"
"DSAIRM_fitbasicmodel"	"fitbasicmodel"	"Basic Virus Model Fitting"	9	"The following point is not directly related to the whole fitting topic but worth addressing: without much comment, I asked you to set the unit conversion factor to 0. That essentially means that we think this process of virions being lost due to entering infected cells is negligible compared to clearance of virus due to other mechanisms at rate _d~V~_. 

While that unit conversion factor shows up in most apps, it is arguably not that important if we explore our model without trying to fit it to data. But here, for fitting purposes, this might be important. The experimental units are TCID50/mL, so in our model, virus load needs to have the same units. To make all units work, _g_ needs to have those units, i.e., needs to convert from infectious virions at the site of infection to experimental units. Unfortunately, how one relates to the other is not quite clear. (See [@handel07] for a discussion of that.) If you plan to fit models to data you collected, you need to pay attention to units and make sure what you simulate and the data you have are in agreement. In general, we might probably want to fit _g_. Here, we briefly explore how things change if it's not 0. Reset all inputs, then set _g=1_. Run a single iteration. You'll find a very poor fit (SSR=786). 

Run solver 2 for 1000 iterations. You'll find something that looks better, but not great. Play around with the starting values for the fitted parameters (_p_/_b_/_dV_) to see if you can get an ok looking starting simulation. This is often required when you are trying to fit, since if you start with parameter values that place the model very far away from the data, the solvers might never be able to get close. Once you have a somewhat decent starting simulation, try the different solvers for different iterations and see if you can improve. One useful approach for fitting is to run for some iterations and use the reported best-fit values as new starting conditions, then do another fit with the same or a different solver. The best fit I was able to find was an SSR of a little bit above 4. You might be able to find something better. Since the fit is not good, it suggests the rest of the model (either the model structure, or choices for fixed parameters) is not good and requires tweaking."	"T9R1"	"SSR after 1000 interations, solver type 2 (round to 2 digits)."	"Rounded_Numeric"	"Round to two significant digits"
"DSAIRM_fitbasicmodel"	"fitbasicmodel"	"Basic Virus Model Fitting"	10	"Keep exploring. Fitting these kind of models can be tricky at times, and you might find strange behavior in this app that you don't expect. Try to get to the bottom of what might be going on. This is an open-ended exploration, so I can't really give you much guidance, other than to re-emphasize that fitting mechanistic simulation models is much trickier than fitting more standard (e.g. generalized linear) models. Just try different things, try to understand as much as possible of what you observe."	"T10R1"	"Nothing"	"None"	""
