ISoP is interested on how our members share models and modeling scripts. The following brief survey will help us understand your current and future plans.
Thanks for setting up this interesting survey. Itâs heartening to see that the majority of responders are happy to share models and code. I think reproducibility and pre-competitive sharing of models and code is vital within Pharmacometrics to help move us forward, stopping âwheel-spinâ (many people developing very similar models for the same endpoint and indication in many different locations). Having the right tools (git or other version control) and a suitable repository for sharing (Github, Stack Overflow, DDMoRe repository, BioModels Database) are essential to make this happen efficiently. While the DDMoRe repository is the ânew kid on the blockâ itâs encouraging to see that modellers are a) aware of it and b) considering using it in the near future.
I agree with you, @MikeKSmith. However, I strongly feel we will be left behind with the thought of just sharing our work as opposed to collectively working together on problems (I am making this statement comparing oursleves to other fields of applied science). We are in the era of networked science and we should strive to tap into each others micro-expertise to facilitate micro-contributions. Such a collective effort goes beyond sharing, but results from the poll are definitely positive in the sense that we are crossing the first hurdle of âintent to share (ITS)â
@ChrissyH Thank you for your message. Please understand that this public forum does not endorse advertisement of products that are closed for public. If you have open source solutions we will be happy to host your messages here. Thank you for understanding â
I definitely agree with the need to be able to collaborate and then share the final scientific product. I think that what we have seen so far here is the overwhelming wish to share (which is the underlying aspect for collaboration) and some aspirational thoughts on how. Time will tell on how we as a community continue to grow our skill sets in collaborative model building, data analysis, and storage of the final product. With the numerous options currently available (and possible future options) we have an exciting future ahead of us.
I think one of the underlying problem areas that must (somehow) be addressed that really rears its head is data sharing. Not that there isnât value in sharing model code, but if the underlying data is locked down, it severely inhibits the ability to discuss and communicate.
Eg, in a âperfectâ world, if you have a problem with a conclusion made from a model, you should be able to run your own analysis through the data to push the discussion forward in a better way. Anyone can criticize a VPC or diagnostic plot, but without a way of being able to work with the data itself, interactive discussion canât really proceed well.
The worst is I canât think of a way to approach the data sharing that companies would buy into - I think blinding is âeasyâ to make it reasonable to share, the tricky part is the collective buy in that trading your potential competitive advantage for that compound by exposing some underlying raw data is less than the overall positive impact on both public health and the rate of progression of drug development for all parties as everyone shares with everyone.
@dpastoor , regarding data sharing, I was wondering if anyone has any experience with Dataverse or any other similar open-source data sharing initiatives with clinical data?
I think there are a few issues at play around data sharing that should be discussed more. (Caveat, these are my own personal views, not necessarily those endorsed by my company or DDMoRe!).
Before we can share any data I believe we need to have informed patient consent to do so. This may be something that is being introduced into patient consent forms, but I think we cannot retrospectively apply it.
Patient identifiability is something that we should certainly take seriously. In some cases it may well be possible to anonymize patient data, but itâs difficult to come up with a single rule or process that will fit everywhere e.g. in rare diseases.
Clearly though, certain agencies, consortia and disease areas have registries and databases already to share data in some way. So it canât be impossible.
Itâs interesting to consider though whether reproducibility applies to a single model / script or to a modelling exercise.
The former means that if I use your code I can reproduce your results, which is essential to ensure that I have all the code and output I need to use your model e.g. to perform VPC or simulation. This would already be a big step forward from code available as supplementary material for manuscripts, or code snippets from forums like nmusers etc.
The latter would entail having the original data so that I can perform my own modelling process and check whether I would come to the same final model as you. The latter is the ideal situation, but experience suggests that for a given set of data, you might well end up with more opinions than the number of analysts looking at the data! MID3 concepts (https://www.ncbi.nlm.nih.gov/pubmed/27069774) suggest that to be transparent we need to share not just run records of key steps, but assumptions and decisions associated with those models. Maybe that level of transparency would enable better critique of models without necessarily having to have the raw data.
Ultimately I think we need to ensure reproducibility at both levels - that I can reproduce the results of any given analysis and that I can track through your thought process and see how you got to the final model.
