UKC

The Sunday Times Good University Guide is out today. Any thoughts, especially if you work in higher education, about which of the streams of information are useful? What relation do they bear to reality? This is from the point of view of potential undergraduates. Presumably, the same points all apply to the Guardian's tables and the Complete University Guide, which seem to work off the same inputs? And - not that it really matters - any idea how their algorithm weights the individual categories to come up with a final ranking? It seems to be a combination of applying random weightings, modified by human interference, if the first pass would result in the universities coming out in a 'wrong' order.

I can tell you for a fact its all total bollocks.

There may be good and bad courses, good and bad teachers, but the concept of the whole university being better than another is far too great a simplification. We looked at a lot of university metrics during teacher training, they all have considerable foibles. However it probably is best to be higher than lower on the tables. 

It might be better if you are looking for a place after COVID to look at

1. Who is going back to face to face teaching?
2. Who is financially stressed?

It amazes me that after years of being told not to destroy people's will to live some folk think that we will not do the same with a Powerpoint on Teams or Zoom and think this is the next great thing. There is a limit to how many Poll Everywhere and Kahoot quizzes you can have and most importantly students need to speak to each other, not just to be sociable, but to learn and that is definitely better face to face.

Absolutely sod all

The exact position on university league tables is almost completely meaningless, as it is mathematical mumbo jumbo: being a combination of dissimilar statistics. Within the categories the methodology degrades statistical meaning, and even on the 'data' there are major problems. 

It's a sign of failure and shame that HE institutions (that should be bastions of truth) play these games. Almost every data area is heavily gamed and even on hard data there are massive mistakes obvious. As an example in the latest Guardian table I was comparing Student Staff Ratio (SSR) on two departments I know well:  one slightly higher staffed with lower student numbers and a good focus on research and another a 'stack them high' place and the SSR numbers look to be swapped (in particular the latter department should have been more than doubled the SSR number given, if staff were counted correctly).

If anyone is seriously looking to go to a particular UK University, consider what life is like outside studies, visit and talk to students, ask questions here (there are quite a few academics and many students and parents). Ignore league tables. Better still go study in the highly rated countries in the EU (on courses taught in English) with lower fees, grant opportunities, cheaper accommodation, a broader experience and better SSR and facilities.

FOR $ in ["Sunday Times Good University Guide", "Complete University Guide", "REF", "TEF", "National Student Survey"] REPEAT:

It's a sign of failure and shame that HE institutions (that should be bastions of truth) play these games [re: $]. Almost every data area is heavily gamed

In my experience (3 kids went/ are at university):

What grade is the child going to get & what's the best place they can go & realistically cope with the level & workload for the course they want to do.

Draw up a shortlist and visit - talk to students. Can the child imagine living there?

Leave the decisions up to them - coach them but don't tell them what to do.

The league tables are obviously bollocks with some great unis not playing the game (E.g. Edinburgh) & others very effective at self-promotion/ gaming the results (E.g. Nottingham Trent, Loughborough).

Very sound advice that, particularly the first point.  I'm not very academic (code for a bit dim) , so got very mediocre grades at A level, but went to an ok university where I really enjoyed myself, coped with the workload and managed to scrape an ok grade in the end....

My brother is much brighter and did a year in industry to wait for a place at a top 5 university but hated it, couldn't deal with the workload and so had to quit the course and start again somewhere else.    

I think TEF needs a special place in hell for extreme cynical bs. REF in contrast IMHO is almost OK but it's way too expensive: in the cost of the exercise itself and the wasted  time of academics (and the extra cost of that...practice exercises...system distortion of extra work  to meet requirements, endless meetings etc). I think replacing panels with metrics would be harder to game, offer much better value for money being way cheaper, and if done well might possibly be more honest than REF.

In reply to blurty

I've never seen Loughborough touted as a massively dishonest gamer before (I know they do game). I think they genuinely do look after their students much better than average. As for the other I'm not allowed to comment but a different midland new Uni were top of my list of executive driven gaming. Top ten and top new uni in the Guardian table for a while despite hearing a lot of serious problems through the grapevine from staff and students.

Does anyone have a link to a non-paywalled list please. Not that it matters now but my daughter is just about to go and I'd be keen on seeing where her uni is placed.

That's not universal. Some institutions at all levels of the system look after students much better than some equivalents and within each institution some departments are much better than others; and then things change over time. I've seen a department go from excellent to poor in a few years with a disastrous change in head. As a broad  historical example, that won't get me sued, in my day (early 80s) Imperial were infamous for driving their elite entry, at times almost to destruction, but Oxbridge were way more student focused for the same courses with the same entry standard.

Potential students need current advice from people with experience there now (or recently).

It's an affront.  An utter affront.

I sort of agree;  the information captured by REF seems pretty reliable, but it still manifests as inappropriate or counter-productive targets applied to research processes and causes its fair share of misery, beyond simply consuming a mind boggling level of resources.

In reply to SimonTp:

I would put the guide to one side and get the kid some boots-on-the-ground time.  How do they feel about the environment, what are the accommodation options like, what do students on the course have to say (we'd run coffee breaks on open day with ~6 open day kids and 1-2 current students per table with a few staff floating about) - is it somewhere they think they'll be happy; with a focus on getting information on the level of support and workload rather than numbers in tables.

Edit: If they don't introduce prospectives to current students on open days, that's notable...  

Yes, agreed!

Even within the same university there can be a lot of variation in what's expected of students.  The university I went to had very different approaches from school to school.  Luckily for me I was in the school of science and engineering where they had absolutely draconian rules, so it made me pull my finger out and get my work in on time (otherwise you got a straight 0%).

Friends of mine in the school of humanities got extension after extension and points deductions left right and centre for late work, and then had to frantically catch up with rushed work art the end of the year.  If I'd been in that school my laziness would have caused me big problems.

Defo sound advice to speak to students there and see how it works, what's expected if you, and like blurty said, can you see yourself there?!  

The problem with REF/TEF/KEF etc is that they consider institutions and departments. However, the work is done by individuals. So when REF comes around then *everything* becomes REF focussed. There's a short gap then the focus shifts onto the next thing. This ignores the fact that's it's the same people doing the same job each time.

I reckon that actual lectures (the non-interactive type with occasional question slots) can work fine online, you could even watch them after the event, but that seminars, tutorials etc are vastly better done in person.

I speak from experience as having done both a "red brick" degree and an OU one - with the OU, the course material (books and online) basically replaces lectures, but other types of session were, when I was doing it, held in person (though I guess there are now Zoom options).

I think your OU experience is different than being in a secure covid group in a hall or house and having six or seven sessions in a day. We record all live lectures anyway for future use, but many still used to come before the current restrictions.

I think it is, yes.  Almost nobody has 6 or 7 lectures a day - they have a mix of lectures, seminars, tutorials and personal study/drinking time (!).  I'm only suggesting that lectures may well work 100% online.

Less satisfying to bunk off a lecture when it's online though!

this . Almost exactly what I was going to put . A friends / neighbours son Is very switched on to this and is applying to various courses in Sweden  , the courses are cheaper , theee are more grants and the towns / city’s are nicer places . He’s already been of his own back and looked at a couple of courses and paid peanuts to fly to those Ryan air type airports know bodies heard of. I reckon this will become  a normal thing to do within a few years  . 

The REF...."almost OK"? It is an apalling measure which - if one adopts the term gaming (sic) - is the most prone of all to manipulation, fad and chumocracy. A loaded dice if ever there was one. Total waste of money as an exercise. It shows how stupid and sycophantic so-called bright people really are.

Well, they were right to say a Scottish University was the best.

But they got the wrong Scottish University, obviously should have been Edinburgh.

Seriously, they have to be smoking something to make St Andrews the best uni in the UK.  My eldest daughter went there for undergrad and then to Edinburgh for her masters.  Edinburgh is miles better.

Thanks, to everyone. None of that is entirely surprising. Open days have migrated online, which makes them convenient but largely useless. Chatting to existing undergraduates does seem useful, though of course the views of people doing the same thing at the same place at the same time can vary, so there's a danger in small samples. You might hope a survey of student experiences was of some use, unless it is completely subject to manipulation and low rates of completion.

In reply to wintertree: As an aside, what combination of incentives and incompetence resulted in a set of evaluative measures that are of so little use (if that is the case)? Do they produce random results or systematically biased results? Who benefits from that mess? 

We have two full days of open days for the department a couple of weeks ago – things are coming on! The students are always roped in to answer questions and as usual it was a great success. 

Depends what you want out of it. Pretty useful if you want to be Queen of Scotland.

Different evaluation frameworks have had different drivers.  My opinionated take below:

REF:  This has a noble purpose in that there should be some accountability for public funds used to pay for academic research. One of the core problems I see is that this works against more blue skies research, and it's pushing things too far towards applications; research is a pipeline and if the inputs (blue skies) are turned off, eventually the outputs (applications) dry up.  The scoring results themselves aren't that bad by general consensus, but the effort consumed in running the process is almost unimaginable, it results in conflicting pressures for management and staff and it's perhaps contributing to problems with the long term viability of academic research.   Who benefits?  In theory the taxpayer benefits by being assured of value for money spent on research...

TEF: The Royal Statistics Society have a lot to say on this - https://rss.org.uk/RSS/media/File-library/Policy/RSS_Evidence_to_the_TEF_co... 

• The RSS still believes that there are many serious statistical problems with the TEF, some of which are around statistical communication. Some of these invalidate TEF.

• Ultimately, the RSS judges it to be wrong to present a provider/subject as Gold/Silver/Bronze without communication of the level of uncertainty. The current TEF presentation of provider/subjects as Gold, Silver, Bronze conveys a robustness that is illusory. A prospective student might choose a TEF Silver subject at one provider instead of a TEF Bronze at another institution. If they had been told that, statistically, the awards are indistinguishable, then their choice might have been different and, in that sense, TEF is misleading.

• TEF Gold at one institution can not necessarily be compared with TEF Gold awarded to another. This is potentially deceptive and misleading for stakeholders, particularly students.

• We are extremely worried about the entire benchmarking concept and implementation. It is at the heart of TEF and has an inordinately large influence on the final TEF outcomes.

I think the REF has been driven by well meaning intent but without bound on the resources consumed and without sufficient thought for unintended consequences.  But for the TEF it's hard to remain so open minded as these comments from the RSS follow previous similar comments raised by them and others in previous rounds of consultation, with little to no visible improvement in the shape of TEF as it emerged from consolation.  I've seen very little that has convinced me that TEF is working in favour of the students in terms of the consequences it has for institutions scared witless about the recruitment consequences the statistically unqualified result could have.  

TEF was so poor because a dishonest government was unwilling to listen to actual evidence and the collective vice chancellors did nothing about that:

https://osr.statisticsauthority.gov.uk/correspondence/concerns-regarding-te...

My concerns on REF are slightly different to wintertree.

My biggest concern by far is cost. The 2014 direct cost was a quarter billion to distribute less than 2 billion. A ridiculous overhead. On top of the direct cost was all the internal preparation and meetings and the work done to produce quite a few otherwise unnecessary outputs that could be measured. In my humble post-92 the staff time alone was way more than any likely income. It's scandalous that no one ever costed this in...research staff time is viewed as an infinitely stretchable piece of elastic 

The second is its divisive steering nature, linked to performance management. Established academics across the world are generally free to explore any avenue they wish but in the UK you need to meet narrower REF targets or your research role will be at risk and in some cases even your actual job and occasionally a whole department might close. As wintertree says this all leads to 'safer' research where outcomes are likely. I'd say its not so much application biased as blue skies light... the worst of both worlds as I think UK research needs all three types for maximum academic and economic benefit.

My third reason is it's soul destroying for many honest researchers because of the wasteful bureaucracy, time and money. Elsewhere great research academics get tenure, better pay and much less of this performance management.

I don't think Greenbanks' view is fair...the previous REF exercises were way more open to gaming and that would be a fair criticism of them but REF 2014, for all its gaming, was a lot more a soundly justified process than TEF.

Disagree.  Definitely getting more out of my online MSc than I ever did for my BSc.  Face to face might work for the more gregarious (i.e. gobby) but probably less so for the less self confident and introverted.  Difficult to be overbearing in the same way on a slack group (for example).

I would ignore it. One of my daughters went to Middlesex which is 123 on the list to study. computer science, she had a habit of being below par in maths , got a great job as software developer.and earns a fortune for a 24 year old working at  a great company.The course was perfect for her.

The other is doing a Masters in Maths at Warwick - now in 4th year and its perfect for her .

So ignore the Guides ( which we did, thank goodness)and go for one that is right for the individual. They are fiddled by the Unis anyway.

Better still go study in the highly rated countries in the EU (on courses taught in English) with lower fees, grant opportunities, cheaper accommodation, a broader experience and better SSR and facilities.

Having recently escaped UK academia for EU academia after about ten years (from undergrad on) I can only agree with this assertion.

As an aside ... the areas my kids have lived in when they went to university (in Leicester, Manchester and Birmingham) have all been squalid - practically 3rd World. It can't be good for mental health or developing citizenship skills.

When we went to visit our daughter at St Andrews she took us for a walk on the beach.   There were students playing polo on the beach.  Polo with horses like in Pretty Woman.

Who TF takes a polo pony to uni?

You always take 2 - never just a pony in case one becomes lame.

Mmmmm.  Does educate them about the harsher realities of life though.

I completely agree regarding the flaws in TEF. The original TQA was at least based on observation of teaching and meetings with students.

As for RAE/REF, the original idea was to allocate QR ££ (i.e. 'blue skies' research ££) preferentially to the departments whose publication records suggested they would make best use of it. That did of course tend to perpetuate a hierarchy, to the detriment of up-and-coming departments, but on the whole it seems more sensible than a flat-rate per-person allocation.

The first few RAEs weren't anything like as time-consuming for everyone as now. The trouble started when the results for each UoA (~ department) changed from a 1 to 5* clasification to a GPA. The media started to publish GPA league tables and university leaders started all sorts of gaming to improve the institution-level GPA. Ironically, in the run up to 2014 I had to point out to the management of my university that their main gaming tactic would actually reduce the eventual QR income, quite apart from all the anguish to academic staff. 

Personally I'd have preferred the Canadian system of individual operating grants, but if we have to have an institution-level exercise I suspect that in many disciplines bibliometrics would give similar results for less than 5% of the effort.

The original TQA was mainly based on a huge quantity of written submissions; there followed a day or so's observations and meetings, as you note.

Since the benchmarks for a good assessment were known long in advance, the submissions were fabricated to meet those requirements. (For example, we had a huge 'staff practices handbook' produced entirely for the occasion. It was binned as soon as the TQA was finished.)

As usual with these kinds of exercise, if the requirements and hurdles are telegraphed in advance, the statistics are manufactured to exceed the requirements. It just like Soviet Tractor Production; and all total bullshit.

<Better still go study in the highly rated countries in the EU (on courses taught in English) with lower fees, grant opportunities, cheaper accommodation, a broader experience and better SSR and facilities>

I have to agree (wholeheartedly), although important to emphasise 'highly rated'

Although Glasgow is obvs better than both of them, because... well... Glasgow!!!!!

You'll probably enjoy giving this a listen https://www.pushkin.fm/episode/lord-of-the-rankings/

Gladwell investigates the US News and World Report rankings of US universities - which seems to be the monopoly ranker for college in the States. What is behind the ranking is pretty ridiculous. I presume it is not dissimilar here.

Oh for sure. Maybe projecting a little, but most of the attractive countries to be in for other reasons *tend* to have highly rated universities.

Lots of universities have polo clubs.  Not many of those playing have their own ponies   kept locally.

Around a billion spent on TQA to find 16 failing departments over the time it ran, most of which were overseas franchise. I worked with one of those overseas colleges and it was OK (and passed comfortably for our arrangement)...TQA failed them for some conflict of interest on the Uni end of the franchise link management, yet the college not the franchising university took the bigger reputation hit. It was way worse than total bullshit, it was also a criminal waste.

In reply to rif

I went through all the changes in REF. In the earlier rounds there were more real issues of nepotistic behaviour and occasionally some utterly bizarre decisions. The time required expanded as much because over-confident departments too often got bad news and VCs forced a massively increased internal focus. It's a bit like the current unsustainable madness in football, more and more resource chases the same limited prizes. I don't care what form of metrics they use just task the quality bodies with being as fair as possible with a capped budget of say £50 million and the worst madness will stop.

Prospective students may ask why is quality assessment stuff relevant to me? It's because part of student fees pays for academic time syphoned off for this. Then those academics distracted by time consuming bullshit have less time and energy for their students. The very best research academics will be tempted to leave the UK or rarely have any significant undergrad contact. More seriously, departments do close due to this bullshit quality measurement and at some point a University will fail... imagine being in the middle of that as a student.

I really hope after all this covid disruption students wake up to the fact that their 'customer contract' with their University is pretty one sided. When things go badly wrong too few seek the compensation they deserve.

I am always intrigued by this view when so many Europeans still come to U.K. universities. Certainly there are lots of French students in U.K. around because they cannot get places in French universities  and so on. 

I don't think that's true as spaces are not limited in France other than getting into the best institutions. I've dealt with various types of French student in the UK. On the success side Erasmus type exchanges normally work very well for French students. As a standard undergrad coming from France it's an expensive but safer way to get to bachelor qualifications, with time also saved if going on to Master's or doctorate level. Then we have French students with parents living and working in the UK, who count as home students.

Its a good way of improving their English

Accents aside, all the french students I knew In my university had better english overall than the average English engineering student! 

Rab C Nesbitt has better english than most gear-heads