Gen Med Project exploring unwarranted variation ready for next phase

The Gen Med Project aims to reduce unwar­rant­ed clin­i­cal vari­a­tion in gen­er­al med­i­cine, using data ana­lyt­ics and machine learn­ing (ML).

Spon­sored and led by Asso­ciate Pro­fes­sor Toby Gilbert, Divi­sion­al Direc­tor of Med­i­cine for North­ern Ade­laide Local Health Net­work, and Tina Hardin, Exec­u­tive Direc­tor, Clin­i­cal Infor­mat­ics and Inno­va­tion (CEIH). The strate­gic and oper­a­tional deliv­ery of the project is pro­vid­ed through the CEIH by project man­age­ment and clin­i­cal infor­mat­ics resources.

The project forms part of the grant pro­gram: SA Acute Care Learn­ing Health Sys­tem Project, coor­di­nat­ed by Health Trans­la­tion SA and admin­is­tered through South Aus­tralian Health and Med­ical Research Insti­tute (SAHM­RI). This pro­gram brings togeth­er health and med­ical research experts to tack­le emer­gency depart­ment over­crowd­ing from dif­fer­ent angles – pre-hos­pi­tal, in hos­pi­tal, and tran­si­tion out of hospital.

It also address­es sys­tem-wide issues that often pre­vent the trans­la­tion of valu­able research into prac­tice or delay it. By using the Monash Health def­i­n­i­tion of a ​‘learn­ing health sys­tem’, the pro­gram hopes to pro­vide strong dig­i­tal foun­da­tions for future projects using machine learn­ing and mod­ern analytics. 

The Gen Med Project began in late 2022 and quick­ly began work to define a clear scope and set of deliverables.

Iden­ti­fy­ing and assess­ing variation 

To iden­ti­fy clin­i­cal vari­a­tion, we must con­sid­er the ​‘what’, e.g. vari­a­tion from what? What are the bench­marks or the stan­dards we are mea­sur­ing against to find vari­a­tion? Once vari­a­tion has been iden­ti­fied, we need to deter­mine what vari­a­tion – if any – is unwar­rant­ed

This clear­ly requires a clin­i­cian dri­ven approach to enable a deep explo­ration into gen­er­al med­i­cine prac­tice. Thus, a work­shop approach was devel­oped, and a phased approach was employed by the project to ensure the right time and atten­tion would be giv­en to the explo­ration before any deci­sions were made regard­ing implementation. 

Data ana­lyt­ics and machine learn­ing (ML)

The project uses sophis­ti­cat­ed data ana­lyt­ics tech­niques to iden­ti­fy vari­a­tion, part­nered with stake­hold­er engage­ment (of clin­i­cians) to assess vari­a­tion. Clin­i­cians iden­ti­fied top­ics, defined para­me­ters, set met­rics and deter­mine out­come mea­sures all towards defin­ing which vari­a­tion is unwarranted. 

ML is not always the answer. The project team used a robust explo­ration process that helps find gen­uine ML use cas­es but fil­ters out (to local teams) any that should be addressed by oth­er means.

Iter­a­tive work­shop­ping and analysis 

CAL­HN, NAL­HN and SAL­HN each took part in a work­shop series start­ing with broad prob­lem and oppor­tu­ni­ty iden­ti­fi­ca­tion. The CEIH team then themed the infor­ma­tion col­lect­ed and start­ed devel­op­ing a ​‘data social­i­sa­tion tool’ — a set of explorato­ry inter­ac­tive dash­boards. This term is used to high­light the pur­pose of the dash­boards: to show what data is avail­able, val­i­date met­rics, explore ques­tions through inter­ac­tive analy­sis, and to sup­port rich­er discussion. 

Addi­tion­al mini-work­shops and larg­er writ­ten analy­ses were under­tak­en on spe­cif­ic con­di­tions, includ­ing heart fail­ure, pneu­mo­nia and chron­ic obstruc­tive pul­monary dis­ease which cov­er a range of ques­tions relat­ing to:

• Dis­ease severity 

• Clin­i­cal out­come measures 

• Inves­ti­ga­tions

• Med­ica­tions

• Fol­low up

This broad engage­ment strat­e­gy has gen­er­at­ed a list of more than 90 topics. 

Where to next 

As we work through the list, we find there are many items that don’t con­vert to a ML use case – and that’s okay! The rea­sons include: 

• Out of project scope

• Insuf­fi­cient data to ful­ly explore

• No vari­a­tion identified

• Vari­a­tion iden­ti­fied is warranted

• Vari­a­tion can­not be defined

• Effort out­weighs benefit

By com­mit­ting only to the oppor­tu­ni­ties iden­ti­fied through our process, we ensure we test top­ics that are worth­while while build­ing the nec­es­sary clin­i­cian buy-in along the way.

As we are in the 5‑year grant peri­od, the project will soon end its explorato­ry phase and focus sole­ly on the hand­ful of ML mod­el oppor­tu­ni­ties that have sur­faced. Work has already begun on a sep­sis pre­dic­tion mod­el, and plan­ning is under­way for a sub-project focus­ing on improv­ing end of life care.

Watch this space!