Tuesday, April 17, 2018

(Some) monks enjoy statistics and other lessons from a recent India trip

Kalden presenting our research in Allahabad
I recently went to India again to continue my collaboration with the monks of Sera Jey monastic university, who are learning to become scientists. This time, we started the visit with the International Conference on Meditation Research, at the University of Allahabad. For the first time, three of the monastic collaborators presented our joint EEG research on Tibetan monastic debate. It was amazing to see them present in a real scientific forum, which was even covered by the Times of India. As one of the monks reminisced, it was also quite historic that the presentation was held in a location quite close to the ancient Buddhist university of Nalanda. The conference also hosted many Indian and international scientists who talked about topics ranging from computational models of meditation (myself) to Hindu philosophy of meditation and its clinical use.


Video scoring in action

Getting our project overview ready


After this exciting meeting I flew to Bangalore, where I gave a few talks, followed by the most important part of the trip: a visit to Sera Jey monastic university. This time, the main goals of our journey were to start a questionnaire study and to improve the rating system of our EEG videos. When we record EEG during a monastic debate, we ask one of the monks to press buttons to indicate events such as agreement, disagreement, distraction, and more. But of course, a problem with this approach is that we do not know how reliable such ratings are. In scientific studies, it is then important to compute inter-rater reliability. We talked about this concept, and then proceeded to re-rate the videos of the EEG sessions we collected, such that it is possible to look at the agreement between raters. This turned out to be quite an interesting experience that led to a lot more discussion between the monks about how to decide when to press a certain button.

Talking about statistics
During this visit, we also had another important realization: that we actually never talked very much about project management. To accomplish a complex team project of the type we are working on, it is crucial to actually divide tasks between individuals, and to have a shared vision on what needs to be accomplished and when. Once we realized this, we put together a big scheme that is now decorating the walls of the Sera Jey science centre, which indicates what the questions are that we are trying to answer, and who is responsible for what part of the work. Such discussions are probably even more necessary when you are working with a multicultural team of Tibetan monks and Western scientists compared to when you are working in a small collaboration with a PhD student at a Western university, because there are just so many more hidden assumptions. Moreover, work cultures are very different, where the West tends to have fixed schedules and predictability, in the Tibetan monastery things happen when they happen, and it is hard to plan ahead (especially given things like power outages).

Despite all this, what was the most surprising thing I learnt this trip? Maybe it's that Tibetan monks like statistics (at least the ones we have been working with). They did not only seem to enjoy geeky conversations during the conference in Allahabad, but also were excited about statistics lessons we ran at the Science Centre.

Friday, March 30, 2018

Open science: do we need that? And if so: how can we get started?

Recently, I organized with the Young Academy of Groningen a workshop on Open Science and Reproducibility. What is open science? Isn't science supposed to be open anyway? Keynote speaker Simine Vazire showed us how this is not always the case. She took us back to the fundamental ideals of science, sharing Merton's norms, which distinguish science from other forms of knowing. Science has a sense of universalism, in which the validity of a scientific claim does not depend on who is making it--there are no arguments from authority. Another ideal is "communality"--the scientific findings belong to everyone and everyone can check them. Science should also be disinterested--all results should be reported without bias. Science should not withold findings that are unfavorable to the scientist. Finally, nothing is sacred in science, and all claims should be tested. But is this really how science proceeds in practice? How many scientific findings can really be checked by everyone? How many scientists are really unbiased? Studies show that even scientists think that most science does not adhere to these ideas. So, we need a change in science to become more open such that it becomes easier for everyone to check scientific claims.
keynote by Simine Vazire

A major problem in science is the emphasis on significant results as a precondition for publication. Unfortunately it is quite easy to obtain significant results with enough p-hacking (trying out many different tests on different subsets of your data) and HARKING (hypothesizing after results are known--presenting the obtained significant results as the original hypothesis). Probably as a result of the commonality of these practices, many studies do not replicate, which was most clearly shown in large-scale attempts at replications (e.g., the Reproducibility Project).
Here I open the workshop

So how can we improve? A good step would be to share all materials and data so others can check it. A good resource recommended was the Open Science Framework. Candice Morey for example has all her materials and data from various projects there. However, during the data management panel the Research Data Management Office mentiond that this does not adhere to the new European regulations on privacy. Better options would be to work with for example Dataverse.nl. Moreover, it is important to really think carefully about you deidentify your data, because with the current machine learning algorithms it is surprisingly easy to identify someone's data by combining a few different sources. Another challenge in opening up your data is that you may not remember the connection between all your graphs and the raw data, or you may feel your data analysis scripts are too messy. Laura Bringmann shared some knowledge about Rmarkdown, which allows you to seamlessly integrate data analysis with code, avoiding the need to have code and graphs and data live in different places. This also makes it really easy to do revisions, because you can easily reproduce the original analyses that lead to specific numbers and plots. Of course even if you decide to open your data, many others may not do so. One practical step individuals can take to enhance openness in science is to participate in the Peer Reviewers Openness Initiative, in which you pledge to only review articles which make their data open (or provide a good excuse why they cannot do so). Of course another way in which openness can be improved is if universities consider the extent to which an individual makes their data and materials open in hiring and promotion decisions.
Candice Morey talking about pre-registration

In addition, what helps to promote open science is to formalize your hypotheses and deposit them somewhere before you collect the data. This procedure is called pre-registration, a topic also discussed in a keynote by Candice Morey. Preregistration can be done quite easily on the Open Science Framework. Another interesting method is to write up your hypotheses in a Registered Report format (offered by more and more journals), in which reviewers decide on acceptance based on your introduction and methods before you collect the data, and then you are guaranteed (in principle) acceptance, independent of how your results turn out. Of course academic incentives should also change to promote this: rewarding these research practices instead of rewarding high-impact publications.

A further step in improving would be to stop overselling our results and better understanding statistics. Rink Hoekstra talked about common misunderstandings about statistics. Most notably, almost everyone's intuitions about p-values are wrong. P-values cannot ever tell you that your statistical hypothesis is true, but it only provides some evidence against a null hypothesis, and it always carries a certain level of uncertainty. It is therefore never possible to make very definite claims about your data, unlike what journals, and even more the media, wants. A very insightful visualization of how little p-values really mean is the Dance of the P-values. Instead of blindly relying on p-values it is critical to instead focus more on visualizing your data, for which Gert Stulp provided some useful resources.
The data management panel

In short, there is still a long way to go to open up your science, but more and more resources are available. The full slides and materials of the meeting can be found here. You can also check out the hashtag #RUGopenScience.


Thursday, February 15, 2018

Working with (against?) work pressure in academia

Today I followed a workshop given by Jeanine de Bruin and Brigitte Hertz on work pressure in academia. This workshop followed at the end of a monthlong writing challenge for academics, in which the goal was to write for two hours every day. I really enjoyed the challenge of consciously shifiting priorities towards generating content and away from responding to things, because even if producing is harder, in the end it is more rewarding (see also my earlier blog post here.

The first insight from the workshop was that the main tool for dealing with work pressure is trying to make the right decisions. It is very easy to get swept away by the onslaught of things to do, and the implicit attitude of "I am busy, so I am working hard and doing it right" instead of thinking about WHAT you are doing.

In managing your work you can think about three levels. The first level is working on a day level. You can start the day with the things that are important to you (e.g., writing this grant application). Moreover, do not have the illusion that you can multitask, because you can really not, and not only the quality of your work suffers, but you also feel much more stressed (at least I do). In addition, I find it very helpful to focus the energy in my day by looking at the goals I defined by means of emacs orgmode; a plain text system in which I stick on my projects with todo items and due dates. This system ensures that no projects are forgotten. I also really like to use it to keep track of each student I am supervising (each student has their own file in which I write a few notes after every meeting with what they have done and what they will do for the next meeting). You can also think about time management at the project level, in which you want to make sure you pick the right projects, and are not afraid to drop projects if they are not leading to something fruitful. Two things I learnt there are first, to take regular space for reflection on how it is going with a project and what is next and so on. This is difficult because the busy woodpecker in me would like to continue doing something "productive", but reflection can really save a lot of time and increase quality in the end. One particular thing to think about are potential roadblocks or plans B: what could go wrong, and what should we do then. I think too little about this, and maybe this wastes a lot of time. On the really long-term level time management is also thinking about what is important for the long-term. What is my passion, where do I want to be 5-10 years from now. I usually only think about this when writing my tenure dossier or a job application, but it is important in deciding what projects to pursue and even what to do in a single day as well. One skilful way of forcing yourself to reflect on this is chatting about it with a mentor over lunch. I highly recommend that. It also made me realize I should review my toggl time log a few times a year to check what my time is being used for, and whether that is what I want. Right now I only review my toggl logs every week when the system mails me the hours tracked.

We then talked about the famous Eisenhower quadrants with urgent-not urgent and import-not important. I have thought about this a lot, and I think that most stuff I do is in the "important" section of the matrix. One thing that I never grasped until this workshop was the distinction between "being important for me" and "being important to do". For example, helping a colleague is a good thing to do, but won't generally help my career ahead. So maybe in the matrix it falls in the category of "not important" in that sense. I realized that I spend too much time following news (partly news, partly news related to science and academic how-to). I made the resolution to limit my news intake to 15 minutes a day, so I won't waste too much time on it. In the end, that won't really get myself too much ahead, apart from staying up to date with the world and getting ideas for lectures, student assignments and grants. Another skilful means for reducing the "not important stuff that needs to be done" is to wonder whether not someone else can do it. I put this into action by requesting a student assistant for the next course I am teaching. While initially this is more work in instructing the person and making all procedures more explicit (grading sheets!), in the end it will save me time. And of course, it is always helpful to remember that saying yes to something means saying no to something else (also described in the book Busy by Tony Crabbe). A final insight from the Eisenhower quadrants was that focusing on the things that are important and not yet urgent will help to increase quality (because you can let the projects rest and then realize you have made some mistakes), and improve your health.

We also discussed what our university can do to reduce work pressure. One important thing I think is that I waste a lot of time searching for information on how to do things, reinventing the wheel, not being aware of procedures. Sharing knowledge more efficiently would really help. We also need to work on more delegation and most importantly better delegation: the insight here was that delegation is not just dropping the task onto someone but also training them how to do it. On the short term training takes time, but on the longer term it reduces errors and therefore headache. It would also be cool if promotion weren't so focused on grant acquisition, because grant acquisition is such a highly risky process that is a lot of work with very uncertain outcomes. I learnt that Utrecht University also has a career path focused on teaching, which sounds really cool. Finally, it would be nice to build more of a collaboration culture at university, rather than the current competition culture.

The last part of the meeting focused on recognizing stress in your work. I think I am blessed in this department because while I feel sometimes exhausted, I never have trouble sleeping and I feel generally happy. I credit the ballet dancing that is a fixture in my schedule, together with the meditation practice that I do every morning and every evening. I think it is really important to stick to those things as priorities in addition to work priorities, because otherwise you waste a whole life working and being unhappy.

In the end, reducing work pressure relies on the confluence of both individual action (trying to work on what is important and inspiring and taking good care of yourself) with institutional action; the universities will have to change because in the long run the current work culture in which many academics put in way too many hours compared to their contracts, and especially in which evaluation is biased towards metrics of grant acquisition that you have little control over is not sustainable. It is helpful to reflect on this both at the individual level and to think together with colleagues and decision makers on how to improve the work culture.

Monday, December 04, 2017

More ideas for activating students, and: should we do that?

Today I had the second installment of the course Flipping the Classroom by Ine Noben of the University of Groningen. It gave me a lot more ideas for activating my students, but also stimulated some interesting thoughts about how we organize our education.
getting creative with lesson design


First, in terms of the ideas: we started with an interesting activity called "concept map", in which you draw a concept map about a topic (in this case the previous lecture), which was a good way to see what we remembered about the previous meeting. We then retrieved even more memories of the previous meeting by "crowdsourcing", in which we walked up to another person in the class, explained the maps to each other, and then exchanged them. When we received them, we scored them on a scale from one to five, and moved on to another person with the new map we got. We continued this a few times and got to see quite a few concept maps. Quite a fun activity!

Another thing we learnt was that it's important to think about how to engage *all* students. That made me realize that when I have some students give presentations, then the others typically disengage and start to play with their phones (or even enter the lecture late). I came up with the idea of asking students to draw a concept map of the presentation and hand it back to me as an 'exit ticket' at the end of class (added benefit: you get a reading of who is present in the class at the same time). Alternatively, I could ask students to write down comments, or things that are incorrect about the presentation, or I could ask them to come up with an exam question about the presentation. Lots of interesting options!

A third interesting activity is having students engage in a debate, where they are randomly assigned to diametrically opposite positions. They have to write a paper about the topic before the class (so they come prepared), and then a subset of students are chosen to debate in class. Of course you have to make sure that all students engage, so you could switch out students at random moment (I can imagine you can make this pretty fun with some weird bell to indicate it's switching time!).

Rink Hoekstra came to share his experiences with the flipped classroom, which he started with the critical note that the flipped classroom is really nothing new, just a repackaging of the older concept of "active learning." He had been experimenting with the flipped classroom (or active learning) and made the interesting observation that students tended to skip the lectures once he provided pre-recorded video lectures because they felt they already knew the information from the video lecture (so what's the point of coming to class?).

We also had some good discussions about whether these active learning forms are not too much like high school. And do they not play too much into the students' extrinsic motivation? On the other hand, how else do you get most students to engage from week 1? An interesting observation by another colleague in this respect was that students really like to see progress, so giving them some form of feedback is crucial.

Finally, we learnt about how video can be a cool tool in the classroom. I am not too excited about recording myself for video lectures, but two things can be very helpful. First, there is a tool called

Monday, November 27, 2017

Invisible scientists and the messiness of science: a dicsussion about open science

Today we hosted a visit of Rosanne Hertzberger with the Young Academy of Groningen. The theme of the afternoon was "open science", and I heard some soundbites that were too good not to share. Rosanne is a very passionate and courageous person who decided to pioneer being a freelance scientist. She started by saying that we as scientists at the university are unaware of how invisible we are. Why? Because we write lots of stuff, that gets put in journals behind a paywall, we talk about our science at conferences that only scientists go to, and we tend to not talk to the public (because we're too busy writing our papers). Good point. Sometimes I feel like the university considers me to be a little hamster running faster and faster in the paper-producing wheel.

She also talked about how science is the only profession where it is not possible to do it as an amateur--you have to be the equivalent of an olympic athlete, or not at all. But why do you have to fully dedicate yourself to science, why is it frowned upon if you have a significant other interest (in her case: writing articles, books and columns). I sometimes feel like that too: why do people think it is so crazy to be a serious amateur ballet dancer as well as a scientist? (see the inspiring quantum physicist Merritt Moore, or my own attempts at dancing and sciencing here). We had a discussion about the extent to which "everyone" can do science (cf. the citizen science movement), but Rosanne retorted that there are so many people who get a PhD and do not get the opportunity to continue in science because there are so few jobs. And another person said: are we even that special as scientists...

Probably one of the most important discussions revolved around the issue of invisibility. Rosanne said "it's very disappointing to see how little openness social media has brought to science. Why is live-tweeting a conference talk still a thing?" In other words, why do scientists not share their talks on youtube? (see for one example to the contrary Richard Morey's periscope broadcasts or the Lab Scribbles open lab notebook). Why don't scientists share their intermediate results on twitter? (while we do see pictures of their kids or cats). We discussed about the benefits of peer review, of which Rosanne posited that it holds us back, because there is too little communication between scientists in the heat of the process about things that work and things that don't. This means that progress is very slow, which is particularly problematic in the case of diseases and epidemics.





I think one other very important point was that in the communication to the public, and in our textbooks, science all looks very clean and shiny, while it is quite messy in the midst of it. Why don't we share our mess online? Rosanne: "it should be standard procedure to overshare. There is no such thing as TMI in science". There was some debate about how this may result in us all drowning in information, but Rosanne argued that a mechanism like reddit would easily allow us to manage this.

A final remark that I really liked was "aren't we reproducing each other's work all the time? It's called scooping." Good point. We ended also discussed quite a bit about incentives in science. Sharing results and materials takes quite a lot of time, for little reward. But this is what will make science progress much more. There is probably also a lot of things we can learn by talking to people from other fields, because in our discussion we learnt that for example in informatics producing reproducible code was standard practice, while sharing event questionnaires is not standard in psychology.

In short: a lot of work needs to be done, and sharing more of our science messiness, materials, intermediate data and so on would probably be a good idea. To be continued!

Monday, November 20, 2017

Activating students: some more tools

Today I took a course about flipping the classroom. Flipping the classroom refers to the idea that students listen to video lectures before class while the teacher helps them with exercises during class (instead of the lecture being spent delivering a lecture while students have to do their homework at home). The most important idea here is that flipping the classroom allows the students to be more active and allows the teacher to help the students where they need help most: to elucidate misunderstandings. In a course by Ine Noben I got some of the following ideas:
Another way to activate students practised for hundreds of years in Tibetan monasteries: requiring them to debate  in pairs while peers are looking on and with large physical gestures.
  1. Get everyone active with 1-2-4-all: students think about a question first alone, then share in pairs, then they share insights with a neighbouring pair and finally you discuss it in the big classroom. Another colleague mentioned that when students are forced to discuss in groups of 4, almost all of them will do some work (because there will always be some active students in the group), especially if you walk around to check on them.
  2. It is important to focus your lecture well, because as H. Simon said "a wealth of information creates a poverty of attention"
  3. A nice blog about peer instruction, with among others a nice series of blogs on why flipped classrooms fail. One thing that can make your flipped classroom fail is if you require students to do all kinds of work, but do not actually give feedback about this work. Another thing that can make your flipped classroom fail is when there are no consequences when students do not do the work, or when students are not clear on what their and your responsiblities are in the classroom. Useful case in point: it's helpful to spell out what you mean by "active participation" in the classroom. I will definitely try this out in my classroom.
  4. Nice platforms for creating online course content: FutureLearn and Google Classrooms. Both allow you to create very nice-looking course materials. Google classroom has the advantage it nicely integrates with googledocs for marking up assignments and things like that. Google classroom also makes it easy to grade with a rubric.
  5. A nice empirical investigation of the flipped classroom: article
  6. Tools for doing online voting: Socrative (allows you to also setup competitions in which teams try to answer as many quiz questions together as possible), mentimeter (allows you to easily vote on single questions and display the results interactively)
  7. A website with active activities: https://compass.itcilo.org as well as a book with game-like activities: Game Storming
  8. If you make quizzes for students to do before class, make sure you give useful feedback, e.g., "go back to section 11.3 of the book"
  9. Tools to make videos: for making screencasts: https://screencast-o-matic.com (e.g., to explain using some data analysis software), or to make cool animated videos/cartoons: https://www.powtoon.com/home/
  10. Another cool tool is "ticket to class"--make sure students do some activity beforehand to earn their ticket to class. One tool I like a lot is Perusall, in which students collaboratively read textbooks or articles. You could also use an exit ticket in which students can only exit the room if they hand in a piece of paper with the answer to a question such as: what is the question still remaining after this class? Name three things that can improve science? Etc (if you require students to add their name to this you immediately have a handy tool to keep attendance).
  11. A very effective tool to get students who are doing poorly back engaged is to find out who is not engaged with learning analytics (checking Blackboard for example) and then sending them a personal e-mail asking why they are not submitting assignments/not showing up etc.
  12. Nice database with exercises for physics, math etc: sowiso
  13. Tool to allow students to make collages of course-relevant information and comment on that (like a kind of instagram): padlet

Sunday, August 27, 2017

Pondering work and meditation

Some reflections from my stay at Ngari Institute of Buddhist Dialectics in Saboo, Ladakh.
Morning story: I woke up to run with the kids at 5:30. Since I wasn't sure where to go, I ran up to what I later found out to be the monk's quarters. Two of the institute dogs came with me. When there was nowhere to go, they guided me back below to where the kids were running. I joined some of them. Together we struggled up the hilll, panting for breath, looking at the beautiful views together, meeting yaks and people on our way. What a beautiful fresh way to start the day!\\
Contemplation:
From Andrew Harvey--''A journey in Ladakh'': ``Those who reject the materialism of the West, who despise it and separate themselves from it, are in danger of refusing to look at it, they are in danger of not being responsible to the facts of life as it is lived, and must be lived, now. We must find a way to work within the world, within science, within industry, even within politics; we cannot simply pretend a superiority to those things, for they are the forces that largely shape mankind. To work within the world we will have to be strong, and in the world our inner strengths will be greatly tested. But that is good. That will dissolve any pride we may have, any sense of virtuous invulnerability. It will take away from us any sense that we are "special'', that we deserve "special treatment'', that we are "unique.''
My continuous struggle is to figure out how to be living a "vita contemplativa.'' What does it mean to be a scientist and yet to also live a spiritual life such that the life and the science are not wasted, but are in fact of benefit to both myself and others. It seems from this writing that what is truly crucial is humility: to just work without expecting anything in return, without any recognition. And also, to drop any thought of being special but just focus on being of help. It is difficult to really live this, because in the West and especially in a career in a competitive profession we are conditioned to continually try to prove we have the best and we deserve recognition. So how can we resist that? Maybe one way would be to contemplate again and again on how we are interdependent with everything. Everything we do cannot be done without the kind help of many others (if only for the food we eat, the electricity we use, and so many more things).
Another helpful quote from the same book: "Be quietly detached from what you do and dedicate it to the good of all created beings, and you will be safe from disillusion or vanity.'' Every time when I go on retreat I realise how important it is to dedicate time in silence. Even just being here, I can see my automatic tendency to ditch meditation in favor of "something useful''--typically processing some information like reading a paper. It is hard sometimes to see the true value of meditation, yet this is the only way we can remain detached from what we do and put it into perspective. And only when we do that can we remain joyful in the face of difficulties, because we see there is so much else that is involved in all of our successes and failures.