Is the randomization of blocks truly random?
Is the randomization of blocks truly random?
have been having some problem with randomization with the Medialab software. I have repeatedly looked over my program, but I can't find any way I could have caused this problem. Of course, randomization problems happen randomly and it could be that, but randomization problems have occurred in two separate experiments, and they are very robust with a p-value of less than .01.
In our first experiment, I had trouble with between group randomization mechanism. THe program was supposed to randomization which of two groups of questions appeared first. One group had 3 question and another group had 2 questions. However 61 percent of the people received the group of 3 questions first and 41 percent received the group of 2 questions first. This difference is significant at p<.01. I have attached the relevant medialab questionnaire. It is called pilot combined. The relevant lines of code are 55 to 59.
In a second experiment, we also experienced randomization problems of a different sort. In that experiment instead of having the computer randomly choose people into conditions, I did it manually using a random number generator. However, I still experienced randomization problems. I had two conditions, in one condition, two scales were correlated at .51 and in another condition they were correlated at -.11, which is p<.001. The problem is that the scales were asked before the manipulation in the experiment. They should have answered the same exact questions up to this point in the experiment. In addition this relationship only exists for one group of people - those who were born in the US. This question was asked during the demographics questionnaire which was placed earlier in the experiment than the two question scales that had randomization problems. (This is an important variable in our theory so that is why this relationship was checked.)
I have attached the code for the complete experiment file and also the single questionnaire (judicial institution) with the two question batteries that had randomization problems. One question battery was composed of the questions from line 14,15,17,18,19, and the other from the question on line 16. I can think of no possible way a programming error on my part or a program bug could conceivably create this relationship, and I have tried everything I could think of to why this relationship would occur. But since it is a 1 in a 1000 chance of it occurring randomly, it is hard to accept that it happened randomly.
I appreciate any help you can give me, and I hope I have been making a simple mistake of some kind and am not just the most unlucky man in the world.
There is a lot here. Lets start with the fact that 61% of your subjects received a "set1-set2" order and 39% (was 41% a typo or am I missing something?) received a "set2-set1" order. You mention that the p level of this is p<.01. Can you tell me a bit more here with regard to how that p level was calculated? esp., with respect to the type of analysis and n-sizes. Any specifics you can provide will help me to understand where the problem could be coming from.
Thanks for the reply. Yes, the 41 percent was a typo. It is actually 39 percent. As for your other questions, the study had 222 subjects, and I calculated the p-value using a simple t test that tested whether the percentage of people in the set1-set2 order was 50 percent. The p-value testing whether I can reject the null hypothesis that the percentage equals 50 percent is .0007.
This is unusual. I checked the pilot combined que and it looks fine in terms of its structure. Although, technically, I think you need to do a categorical/binomial analysis (chi-square, logistical regression? a quick java applet for such is here: http://stattrek.com/Tables/Binomial.aspx) rather than a t-test, the probability is still going to make it very surprising to get 88/222 assigned to one of two combinations. Hmm.
We have been using the same BGR randomization routines in MediaLab for almost 14 years now and I've never heard of a report like this so I don't know what to say. If it were me, I would run a simulation to see if I could replicate the problem--e.g., delete all but the essential items from the pilot questionnaire and then run the shortened (maybe 10 item revision) twenty or thirty times and see if the distribution is off balance in any predictable way. If it DOES recur then please please let me know.
