Gen Med Project exploring unwarranted variation ready for next phase.

The ​‘Gen Med Project’ aims to reduce unwar­rant­ed clin­i­cal vari­a­tion in gen­er­al med­i­cine, using data ana­lyt­ics and machine learn­ing (ML).

Spon­sored and led by Pro­fes­sor Toby Gilbert, Divi­sion­al Direc­tor of Med­i­cine for North­ern Ade­laide Local Health Net­work, and Tina Hardin, Exec­u­tive Direc­tor, Clin­i­cal Infor­mat­ics and Inno­va­tion (CEIH). The strate­gic and oper­a­tional deliv­ery of the project is pro­vid­ed through the CEIH by project man­age­ment and clin­i­cal infor­mat­ics resources. 

The project forms part of the grant pro­gram: SA Acute Care Learn­ing Health Sys­tem Project which is coor­di­nat­ed by Health Trans­la­tion SA and admin­is­tered through South Aus­tralian Health and Med­ical Research Insti­tute (SAHM­RI). The pro­gram sees a col­lec­tive of health and med­ical research experts band togeth­er to tack­le the issue of over­crowd­ing in our emer­gency depart­ments, from var­i­ous lens­es – pre-hos­pi­tal, in hos­pi­tal, and tran­si­tion out of hos­pi­tal. The pro­gram also address­es sys­tem-wide issues that often pre­vent the trans­la­tion of valu­able research into prac­tice or delay it. By adopt­ing the Monash Health def­i­n­i­tion of a ​‘learn­ing health sys­tem’, the pro­gram hopes to pro­vide strong dig­i­tal foun­da­tions for future projects util­is­ing machine learn­ing and mod­ern analytics. 

Kicked off for­mal­ly in late 2022, the Gen Med Project com­menced plan­ning, and very quick­ly began work to define a clear scope and set of deliv­er­ables for the project. 

Iden­ti­fy­ing and assess­ing variation 

To iden­ti­fy clin­i­cal vari­a­tion, we must con­sid­er the ​‘what’, e.g. vari­a­tion from what? What are the bench­marks or the stan­dards we are mea­sur­ing against to find vari­a­tion? Once vari­a­tion has been identified, 

We need to deter­mine what vari­a­tion, if any, is unwar­rant­ed.

This clear­ly requires a clin­i­cian dri­ven approach to enable a deep explo­ration into gen­er­al med­i­cine prac­tice. Thus, a work­shop approach was devel­oped, and a phased approach was employed by the project to ensure the right time and atten­tion would be giv­en to the explo­ration before any deci­sions were made regard­ing implementation. 

Data ana­lyt­ics and ML 

The project uses sophis­ti­cat­ed data ana­lyt­ics tech­niques to iden­ti­fy vari­a­tion, part­nered with stake­hold­er engage­ment (of clin­i­cians) to assess variation. 

Clin­i­cians iden­ti­fied top­ics of explo­ration, defined para­me­ters, met­rics and, out­come mea­sures all towards ulti­mate­ly defin­ing which vari­a­tion is unwarranted. 

ML is not always the answer, and so the project team was com­mit­ted to employ­ing a robust process of explo­ration that would not only sur­face gen­uine ML use cas­es, but fil­ter out (to local teams) any that should not be addressed through ML. 

Iter­a­tive work­shop­ping and analysis 

CAL­HN, NAL­HN and SAL­HN were each offered a work­shop series start­ing with broad prob­lem and oppor­tu­ni­ty iden­ti­fi­ca­tion. The CEIH team took this away and themed the infor­ma­tion col­lect­ed and set to work on devel­op­ing a ​‘data social­i­sa­tion tool’, which is anoth­er term for set of explorato­ry inter­ac­tive dash­boards. We use this lan­guage to high­light the pur­pose of the dash­boards, which is to high­light what data is avail­able, val­i­date met­rics, explore ques­tions through inter­ac­tive analy­sis, and to facil­i­tate and enrich discussion.

In addi­tion, the project team deliv­ered a mini work­shop series to address ques­tions relat­ing to heart fail­ure, and have embarked on larg­er, writ­ten-report-style analy­ses for things like pneu­mo­nia care and chron­ic obstruc­tive pul­monary dis­ease, which cov­er a range of ques­tions relat­ing to: 

• Dis­ease severity 

• Clin­i­cal out­come measures 

• Inves­ti­ga­tions

• Med­ica­tions

• Fol­low up. 

This broad engage­ment strat­e­gy and breadth of explo­ration tech­niques has led us to a list of over 90 top­ics in total. 

Where to next 

As we work through the list, we find there are many items that don’t con­vert to a ML use case – and that’s okay! The rea­sons include: 

• Out of project scope. 

• Insuf­fi­cient data to ful­ly explore. 

• No vari­a­tion identified. 

• Vari­a­tion iden­ti­fied is warranted. 

• Vari­a­tion can­not be defined. 

• Effort out­weighs benefit. 

Com­mit­ting to only the oppor­tu­ni­ties that sur­face through our process means we will only test the top­ics that are worth test­ing, and, by tak­ing the process we have we’re build­ing the nec­es­sary clin­i­cian buy-in along the way. 

As we are in the 5‑year grant peri­od, the project will soon end its explorato­ry phase and focus sole­ly on the hand­ful of ML mod­el oppor­tu­ni­ties that have been surfaced. 

We are already work­ing on a sub-project for a sep­sis pre­dic­tion mod­el and are in the defin­ing stages for a sub-project focus­ing on improv­ing end of life care. 

Watch this space!